ARC Accounting

ARC is keeping an account of computing resources usage, and this activity covers all jobs from all ARC users.  This page explains
  • what "usage" means;
  • how ARC accounts for that usage;
  • how we automatically add credit to project accounts

How Credits are Managed

By default, users consume credits from a pool belonging to their project when they run jobs.

In the past, users had to check how many credits they had available and if their credit was zero or negative, their queued jobs would be held and they would have to request additional credit to ensure their jobs could continue to run. 

From 01/09/2022 we will ensure all projects have sufficient standard credit at all times.

This means that we will automatically add Standard Credit for all projects in the Mathematical, Physical and Life Science (MPLS), Humanities (Hum), and Social Sciences (SSD) divisions which are subscribed to the Divisional Funding model. Medical Sciences Division (MSD) does not currently subscribe to the Divisional Funding model and projects in this division will continue to have access to basic level credit.

More detailed explanation of Standard and Basic Credit is available on our web pages, ARC Service Level Agreement .

The ARC Accounting Method

ARC uses an accounting package called GOLD. This is coupled with some homegrown extensions and hooks into the job schedulers. A job that runs on ARC systems use up "credits". A credit is equal to running 1 processing core for 1 second of wall clock time, and is independent from how fast the processing core is. An alternative way to express usage is to use "core hours"; 1 core hour is the equivalent of 1 processing core being used for 1 hour of wall clock time, so:

3600 credits = 1 core hour

The amount of credits that are "consumed" by a single job depends on:

a) the amount of resources that are allocated to a job and

b) the duration of the job.  

The amount of credits which a job consumes are then calculated thus:

 consumption = (duration of job) x ((number of nodes allocated to the job) x (number of processor cores requested))

The duration of the job is the real, "wall clock" duration, and not CPU time as measured by the operating system.  The duration of a job is measured to the nearest second.  The compute nodes on the two main ARC systems (ARC and HTC) typically have a maximum of 48 processor cores in each compute node which can be requested by your job.

If for example, a job ran for ten hours on one full compute node of ARC, the number of credits consumed would be:

(10 X 3600)  x (1 X 48) = 1,728,000 credits

Please note: If you use exclusive allocation in your job, the same number of credits would be charged regardless of whether all 48 cores or a single core was used.

Some applications are simply serial (1 single process with 1 thread of execution) and cannot be changed.  In that case, in order to utilise the cores available on a node, it is possible to "pack" more than one process of the same application inside a single job executing on a single node.  This mechanism is described here, and works extremely well to parallellise workloads in which the same application is mapped to a large number of cases, e.g. a Monte-Carlo simulation or a parameter sweep.

Monitoring Usage

Users can use sacct , provided by the SLURM job scheduler to print information about jobs.  To find elapsed time for a job you are running, you can use the command:

sacct --job=jobnumber --format=JobID,elapsed

The value printed by sacct has to be converted to hours by dividing by 3600.

 

GPU Accounting

GPU processing is charged at the rate of 8 normal CPU cores each. For example running on a single GPU for ten hours would consume:

(10 X 3600)  x (1 X 8) =  288,000 credits

Requesting more ARC Credit

Requesting credits for the free service

Please email support@arc.ox.ac.uk with a conservative estimate of the number of credits or CPU hours you require to run your jobs, along with a brief description of the research project that ARC will be supporting. One of the team will respond promptly and add the credits to your ARC project.

Free service equates to 'standard' credits for ARC Projects within MPLS, SSD, or Humanities Division, and 'basic' service credits for MSD projects.

Purchasing credits for enhanced Quality of Service

Enhanced QoS is 'priority' for project within MPLS, SSD, or Humanities Division, and 'standard' for MSD projects.

Procedure

If you would like to purchase credits with funds from a research grant please contact us first to discuss your requirements.

Once this is done and you know how many credits to purchase, please complete the Internal Trade Recharge Form using either the FEC or non-FEC options and sent by email to support@arc.ox.ac.uk or in hard copy to Advanced Research Computing, IT Services, 13 Banbury Road. Alternatively, payment can be made via Purchase Order on request.

Once the payment is received, you are notified that the credits purchased were added to the project.

Cost of Credits

Pricing is reviewed annually. Compute credits bought do not expire.

The Full Economic Cost (FEC) of credits for CPU compute time is £0.01 or 1p per core hour (e.g. 10,000 core hours will cost £100). For GPU accounting this CPU core hour charge is increased to £0.08 or 8p, with each additional GPU requested costing an additional £0.08 per CPU core hour used.

You should use this cost in any grant applications, this is based on ARC being an SRF type of facility.

The cost of additional storage allocation above the standard 5TB per project allocation is £250 per TB per year. This cost is on top of that of compute time.

If you require costings for EU grant applications please contact us.

Purchasing credits with grants and other sources

Purchasing credits with an FEC Grant

Typically, this will be from a specific Directly Allocated Facility cost from a Research Council or similar type of grant under Full Economic Costs (FEC). For FEC grant you will need to provide the following:

  • Cost centre and account string you wish to be debited
  • Line description for the journal
  • PI (budget holder) signature
  • Departmental Adminstrator's signature

Non-FEC Projects and other sources

In the case of non-FEC grants you will need to provide the following on the form:

  • Project code
  • Task
  • Award number
  • Expenditure type

FP7 EU Funding (Directly Incurred headings)

Owing to audit requirements we have to handle such funding in a different way. Essentially we have to invoice the user in arrears for actual, auditable usage. We can do this but it's best to get in contact first to discuss.

External Users

We welcome external users; please contact us to discuss your requirements.