Data Analysis

Access CXLS Docs Access CXFEL Docs

Contents:

• Quick Start Guide for Data Analysis at CXLS
  • Getting Started
  • Setting up the Data Stream
  • Accessing the Data
• Access
  • AGAVE Access
  • Sol Access Page
  • Globus File Sharing
• AGAVE Access
  • Useful References
  • Prerequisites
  • AGAVE Login Steps
  • Loading Modules
  • Partitions at AGAVE
  • Running pattern_sim through CrystFEL
  • Indexing Images indexamajig through CrystFEL
• Sol Access Page
  • Introduction
  • Access Methods
  • User Guide
  • Additional Resources
  • Module Load Commands
• Globus File Sharing
  • Run Application (with GUI)
  • Run Application (No GUI)
  • Globus Setup in Command Line (No GUI)
  • Web Application
  • Accessing AGAVE with Globus
• Submitting SLURM Jobs
  • AGAVE indexamajig Script
  • Parameters
  • Script Description
  • Usage
  • AGAVE pattern_sim Script
  • Usage
  • Parameters
  • Script Description
  • Key Components
• External Software
• Custom Software
  • RCSB Search API Scripts
• RCSB Search API Scripts
  • Official Documentation
  • Installation and Setup
  • Accessing PDB IDs using scrape.py
  • Downloading PDB Files using download_pdb_files.py
• Online Analysis
  • Introduction
  • Online Analysis Tools
  • Online Analysis Workflow
  • CrystFEL
  • Reborn
  • AI/ML Data Package 1 (gpCAM)
  • AI/ML Time Resolved Data Package
  • Conclusion
• Offline Analysis
• Projects
  • Water Background Subtraction Project (Main)
  • Water Background Subtraction Project (Stream)
  • Water Background Subtraction (Overwrite)
• Water Background Subtraction Project (Main)
  • Imports
  • PeakThresholdProcessor Class
  • ArrayRegion Class
  • Helper Functions
  • Coordinate Menu Function
  • Main Function
• Water Background Subtraction Project (Stream)
  • Imports
  • PeakThresholdProcessor Class
  • ArrayRegion Class
  • Helper Functions
  • Coordinate Menu Function
  • Load Stream Function
  • Main Function
• Water Background Subtraction (Overwrite)
  • Imports
  • Duplicate Function
  • Debugging Functions
  • Overwrite Function
  • Intensity Finder Function
  • Populate Intensity Array Function
  • Main Function

Introduction

Here is the comprehensive homepage for the CXLS/CXFEL data analysis. This page will describe the data analysis workflow for CXLS and CXFEL.

If you are looking for a quick start guide, please see the Quick Start Guide.

Fundamentally, the workflow is divided into two parts: online and offline analysis. Online analysis is performed during the experiment, while offline analysis is performed after the experiment. The online analysis is performed on the fly, and the results are used to make decisions about the experiment. The offline analysis is performed after the experiment, and the results are used to publish the data.

Because of the size of the data being transferred, we have to seperate this into two parts.

1. Online Analysis: During the experiment, we will stream the data directly to the data analysis station for quick viewing. By this workflow, quick changes can be made to obtain the desired images.

   This process will be further described here: Online Analysis

2. Offline Analysis: After the experiment, this data is then stored by ASU Research Computing. The experiment data will be stored here for further processing.

   This process will be further described here: Offline Analysis

Note

The data analysis station is a Linux machine.