ARC Accounting

ARC is keeping an account of computing resources usage, and this activity covers all jobs from all ARC users.  This page explains

  • what "usage" means;
  • how ARC accounts for that usage;
  • how usage relates to charging and
  • how system time is allocated to projects and users.
  • how to request more credit.

How Credits are Managed

By default, users consume credits from a pool belonging to their project when they run jobs. Every user can check how many credits they have available with the mybalance command.  Some typical output of this command is shown below:

prompt> mybalance
Please wait: Calculating balance ...
Your are a member on the following project(s): dept-proj
and your total balance is: 999871360 credits (~277742 hours)
 
Detailed account balance:
Id  Name      Amount         Reserved    Balance       CreditLimit     Available
--  --------      -----------         -------------    -----------       --------------     ------------
51 dept-proj 999871360    0                 999871360   0                   999871360

In the above example, the user can access the credits of the "dept-proj" ARC account, to which the user belongs. The amount of credits available to the project is retrieved from the accounting system and printed.

Another useful command is sacct, provided by the Slum job scheduler to print information about jobs.  To find elapsed time for a job you are running, you can use the command:

sacct --job=jobnumber --format=JobID,elapsed

This number printed by sacct has to be converetd to hours (divide by 3600).

What to do when credits are exhausted?

Every new project on ARC is given an initial amount of credits, worth 25,000 core-hours of computing.  While this is normally sufficient for testing and benchmarking an application, and even do a few production runs, it usually prove not enough for carrying out all the intended computational work.  Individual users (or the ARC account holders) can request additional credits at any time; please refer to the Requesting Usage Credit section for details on how to request more credits.

The ARC Accounting Method

ARC uses an accounting package called GOLD. This is coupled with some homegrown extensions and hooks into the job schedulers. A job that runs on ARC systems use up "credits". A credit is equal to running 1 processing core for 1 second of wall clock time, and is independent from how fast the processing core is. An alternative way to express usage is to use "core hours"; 1 core hour is the equivalent of 1 processing core being used for 1 hour of wall clock time, so

3600 credits = 1 core hour

The amount of credits that are "consumed" by a single job depends on a) the amount of resources that are allocated to a job and b) the duration of the job.  

The amount of credits which a job consumes are then calculated thus:

 consumption = (duration of job) x ((number of nodes allocated to the job) x (number of processor cores per node))

The duration of the job is the real, "wall clock" duration, and not CPU time as measured by the operating system.  The duration of a job is measured to the nearest second.  The compute nodes on the two main ARC systems (Arcus-B and Arcus-A) have 16 processor cores in each compute node.

If for example, a job ran for ten hours on one compute node of Arcus-B, the number of credits consumed would be:

(10 X 3600)  x (1 X 16) = 576,000 credits

To reiterate the important point about exclusive allocation, the same number of credits would be charged regardless of whether a single core or all 16 were requested.

Some applications are simply serial (1 single process with 1 thread of execution) and cannot be changed.  In that case, in order to utilise the cores available on a node, it is possible to "pack" more than one process of the same application inside a single job executing on a single node.  This mechanism is described here, and works extremely well to parallellise workloads in which the same application is mapped to a large number of cases, e.g. a Monte-Carlo simulation or a parameter sweep.

GPU Accounting

GPU processing is charged at the rate of 8 normal CPU cores each.  This results from the fact that there the GPUs on Arcus-B are 2 on a compute node, and each node has 16 cores in total.  Therefore, using one GPU is considered equivalent to occupying 8 cores for the duration of the GPU processing.

For example running on a single GPU for ten hours would consume:

(10 X 3600)  x (1 X 8) =  288,000 credits

Requesting Usage Credit

Requesting credits for the free standard service

Please email support@arc.ox.ac.uk with a conservative estimate of the number of credits or CPU hours you require to run your jobs, along with a brief description of the research project that ARC will be supporting. One of the team will respond promptly and add the credits to your ARC project.

Purchasing credits for the priority services

Procedure

If you would like to purchase credits with funds from a research grant please contact us first to discuss your requirements.

Once this is done and you know how many credits to purchase, please complete the Internal Trade Recharge Form using either the FEC or non-FEC options and sent by email to theteam@arc.ox.ac.uk or in hard copy to Advanced Research Computing, IT Services, 13 Banbury Road. Alternatively, payment can be made via Purchase Order on request.

Once the payment is received, you are notified that the credits purchased were added to the project.

Cost of Credits

Pricing is reviewed annually. Compute credits bought do not expire.

The Full Economic Cost (FEC) of credits for CPU compute time is £0.01 or 1p per core hour (e.g. 10,000 core hours will cost £100). For GPU accounting this CPU core hour charge is increased to £0.08 or 8p, with each additional GPU requested costing an additional £0.08 per CPU core hour used.

You should use this cost in any grant applications, this is based on ARC being an SRF type of facility.

The cost of additional storage allocation above the standard 5TB per project allocation is £250 per TB per year. This cost is on top of that of compute time.

If you require costings for EU grant applications please contact us.

Purchasing credits with grants and other sources

Purchasing credits with an FEC Grant

Typically, this will be from a specific Directly Allocated Facility cost from a Research Council or similar type of grant under Full Economic Costs (FEC). For FEC grant you will need to provide the following:

  • Cost centre and account string you wish to be debited
  • Line description for the journal
  • PI (budget holder) signature
  • Departmental Adminstrator's signature

Non-FEC Projects and other sources

In the case of non-FEC grants you will need to provide the following on the form:

  • Project code
  • Task
  • Award number
  • Expenditure type

FP7 EU Funding (Directly Incurred headings)

Owing to audit requirements we have to handle such funding in a different way. Essentially we have to invoice the user in arrears for actual, auditable usage. We can do this but it's best to get in contact first to discuss.

External Users

We welcome external users; please contact us to discuss your requirements.