CRSP Recover From Accidentally Deleted Files

This guide will tell you how to find and copy a deleted file from a snapshot using different access methods.

1. Snapshots

1.1. What is a "Snapshot"?

A snapshot of a file system is a logical, point-in-time, read-only, copy of all files. It’s not really a complete copy. Instead, the file system keeps track of files that are changed or deleted after the snapshot was made. By definition, all snapshots are read-only, meaning you cannot delete a file from a snapshot. Restoring a file from a snapshot is as simple as copying the file back to your working directory/folder.

On CRSP, all snapshots are labeled by date and time. The timezone is GMT (Greenwich Mean Time). Snapshots are point-in-time copies of the CRSP file system. Snapshots are taken

  • daily, keep last 14

  • weekly, keep last 8

1.2. Is Snapshot a Backup?

Not really. Backups are generally thought of as historical copies of files and users could go to a backup to recover a file from many months ago. Snapshots provide some safety against the common "accidentally deleted" use case. Files created and deleted in the same time interval between two snapshots are not recorded in any snapshot and have no recovery. CRSP does not keep historical backups of data. Files that were deleted more than 8 weeks ago are gone forever.

2. Finding Snapshots

Due to the way the underlying filesystem (GPFS) is architected, you must first navigate to the "top level" of the CRSP file system and then navigate downwards to the correct snapshot folder to find your system.

This means that you will see all possible labs or home area folders (and there are 1000s of them on CRSP). Rest assured that only you and those you designate can see the appropriate files. All access permissions are fully enforced, even when navigating snapshots.

The snapshots are held in the folders HOME-SNAPSHOTS and LAB-SNAPSHOTS. Each snapshot is folder that is named after its creation date.

2.1. From HPC3

The snapshots are available is in /share/crsp/HOME-SNAPSHOTS and /share/crsp/LAB-SNAPSHOPTS.

For example, a user panteater can find what are the snapshots of its HOME area as:

[user@login-x:~]$ ls /share/crsp/HOME-SNAPSHOTS/
@GMT-2021.07.11-10.00.00  @GMT-2021.08.06-01.00.14  @GMT-2021.08.10-13.00.07
@GMT-2021.07.18-10.00.00  @GMT-2021.08.07-01.00.14  @GMT-2021.08.11-01.00.14
@GMT-2021.07.25-10.00.00  @GMT-2021.08.08-01.00.14  @GMT-2021.08.11-13.00.07
@GMT-2021.08.01-10.00.00  @GMT-2021.08.08-10.00.00  @GMT-2021.08.12-01.00.14
@GMT-2021.08.03-01.00.14  @GMT-2021.08.09-01.00.14  @GMT-2021.08.12-13.00.07
@GMT-2021.08.04-01.00.14  @GMT-2021.08.09-13.00.07  @GMT-2021.08.13-01.00.14
@GMT-2021.08.05-01.00.14  @GMT-2021.08.10-01.00.14  @GMT-2021.08.13-13.00.07

And then browse the contents of a specific snapshot as :

[user@login-x:~]$ ls /share/crsp/HOME-SNAPSHOTS/@GMT-2021.08.08-10.00.00/panteater
Use your UCnetID after the specific snapshot name.

2.2. From CRSP Desktop

You can navigate in CRSP Desktop to the top-level by connecting to the crsp-top-level share connection that is predefined by the UCI installation of CRSP Desktop. See Windows or Mac instructions. Once at the top level, you will find snapshots labeled by their creation date in the folders labeled LAB-SNAPSHOTS and HOME-SNAPSHOTS.

2.3. From web browser

You can also find the top level of CRSP while using the web browser interface to CRSP. When you navigate to the CRSP top level, you will see a folder structure that is similar to the following:

crsp lab top level
Figure 1. CRSP FIle Browser Top Level.

Snapshots are held in the folders labeled LAB-SNAPSHOTS and HOME-SNAPSHOTS. If you click on LAB-SNAPSHOTS, you will see a set of folders that are named after their creation date:

crsp lab snapshots
Figure 2. Time stamped Snapshots in the LAB-SNAPSHOTS folder

In this example, the most recent snapshot is @GMT-2021.05.05-19.00.1. Its name indicates the time stamp when this snapshot was taken: May 05, 2021 just after 7PM (GMT). This translates to 11AM (PST). This folder contains that logical copy of all CRSP lab folders, as they were at that point in time.

3. File Recovery From a Snapshot

3.1. From CRSP Desktop

Use your CRSP Desktop application to connect to the desired share (Windows or Mac instructions) then use it just like a folder or network drive to copy files and folders from a specific snapshost folder.

3.2. From HPC3

Once you found the snapshot that holds your desired files and directories can use usual commands ls, cd, cp etc. to find and copy your desired files and directories from the snapshot to the location where you need to restore them.

For example, a user panteater can do the following to restore a single file accidentally deleted from HOME area:

[user@login-x:~]$ cd /share/crsp/home/panteater
[user@login-x:~]$ cp /share/crsp/HOME-SNAPSHOTS/@GMT-2021.08.08-10.00.00/panteater/important-file important-file

3.3. From web browser

In order to recover the file, you must navigate to the top level directory and then into the lab snapshots.

At this point, you want to select the folder that has a (backup) copy of your file. In this example, we are navigating to the @GMT-2021.05.05-19.00.1 folder. Please notice in the following screenshot that the path is LAB-SNAPSHOTS > @GMT-2019.5.13-19.00.1. This indicates that we are in a snapshot copy. We will continue to navigate into the correct sub folder that holds a copy of file module-hpc.log-20201011.

After the desired files are found, simply choose and download them:

crsp lab snapshot file
Figure 3. Labs that are part of a particular snapshot

and then upload the downloaded files to your desired writable folder. You may also copy the file in your usual manner per your host operating system (Windows) (Mac) (Linux).

At that point, you have restored from the snapshot your desired files.

Please see guides below that provide more information and explain how to get an account, use CRSP and get help:

CRSP Howto Guide

Troubleshooting CRSP Access

Frequently asked questions

LAB allocation

File recovery

Recharge

CRSP Access

How CRSP is built