Fixity Checking

Note: in order to receive email reports, you must subscribe to SNS topic fixity in US-West-1. Go to the topic, create subscription, fill in your email and respond to the confirmation email you will get. 

Our serverless fixity program kicks off as an S3 batch job that invokes lambda function fixity check, which in turn calls the Serverless Fixity step machine. Results are left in the cor-devops-binary bucket. When AWS events detect a new report created,  lambda function fixity-sns-notification is invoked, which generates a new SNS message in topic fixity checks. Any subscribers to this topic receive an email with a link to the results report. 

Note that any errors will reside in a second report, so that subscribers receive up to 2 emails per fixity job run.

If you only receive one notification email, make sure to double-check the results folder to see if a failures report has been generated.

To invoke a fixity check log into the console at aws.emory.edu and select account 70. Navigate to the S3 service menu and select batch operations. 

From the Batch operations menu select Create Job.

Step 1 Choose Region and Manifest

The region MUST be US-east-1

The manifest file describes which objects in the bucket to run the job on.

Complete inventories of the fedora-cor-binary buckets are regularly generated and stored in bucket cor-s3-inventory-2.

For the backup/second copy bucket, these are stored in cor-s3-inventory-3. 

S3 Buckets > cor-s3-inventory-3 > backup-fedora-cor-binaries/ inventory-report/

Select the most recent manifest.json file for the appropriate bucket.

As a shortcut, you can type in a month and day in the search box, e.g. 2024-02 to see the current month's manifests.

You may also generate a csv file with any subset of a bucket's objects and upload that to an s3 bucket.  That csv file must have 2 columns containing the bucket holding the object in the first column, and the object name in the second, no column headers. 

When preparing a custom manifest, upload your csv batch file to the following location: 

cor-devops-binaries/fixity-check-manual-batches/

You can then use the Browse S3 option later to select this file.

Step 2 Choose Operation

Select Invoke Lambda Function and fixity-check-s3-bucket

If you see a prompt regarding Lambda invocation version schema, make sure Version 1.0 is selected (which it should be by default).

Step 3 Configure additional Options

Select generation completion report and specify s3://cor-devops-binaries/fixity-check-results/ and either the fedora arch or fedora prod folder as appropriate. 

use s3-batch-job-role from existing IAM roles

Use tag with name EmoryApplicationName and value fixity

Review and confirm choices in the final step.

Once you create the job, you will need to wait for the job to be prepared and reach status "awaiting your confirmation to run." Make sure that Total objects listed in the job is the number of objects you expect. Then select the job and hit run. Jobs may take multiple hours! You will receive 1 or 2 emails on completion with links to generated completion reports.