2.2. Process Experimental Data

Note: make sure you have configured the directory for the repository and the datasets correctly! (see introductory notes)

This notebook illustrates how one can process the data.

2.2.1. Preparation

We begin by importing the necessary packages. To avoid cluttering, we suppress all warnings for the moment.

import os
import numpy as np
import warnings
warnings.filterwarnings('ignore')

import lib.utilities as ut
from lib.constants import *
from lib.config import DATA_DIR
from lib.data_handling.mf_parser import process_batch

We extract the root directory of the repository and set the base directory of the processing routine to the ~/datasets/MINFLUXStatic/ directory to analyze MINFLUX measurements of static systems.

base_in = os.path.join(DATA_DIR,'MINFLUXStatic/')

2.2.2. Process the data

Last, we specify the parser type (in this case “otto”, since this data was obtained by my colleague Jan Otto Wirth). Depending on the setup that was used to obtain the respective data, we have to deal with different formats and thats why we tell the machine what kind of data to expect. We also pass a dictionary of instructions to the routine, which specifies how the data should be analyzed. In this case we want to chunk our traces in packages of 5000 or 1000 photons, and allow a maximum of 10 chunks per trace (otherwise it would take too long…). You could also choose a different chuinking method, e.g. 'tuple', which splits the trace into chunks of equal number of measurement tuples (important if you want to have equal time bins for each chunk!) etc. Please consult the api documentation for more detailed information.

In order to save some time, we use pre-parsed and pre-filtered files and just do the processing to obtain distance estimates. Running this script may take several minutes (depending on the available computation power and number of files chosen for the analysis), since we are processing quite an amount of data. Of course you can run the parser and filter routines yourself and play around with the parameters. Consult the API documentation for details of the available classes and methods to do that.

Results will be saved to your OUTPUT_DIR directory, which you have configured in lib.config

Note: I have set the ``max_files`` parameter to 10, use ``max_files=np.inf`` in order to process all available files.

parser_type = 'otto'

bootstrap_dicts = [
    dict(mode='photons', chunk_size=1E3, overlap=0., bin_size=1E3,max_chunks=2)|dict(estimator='min-poly', plot=False, output=None)
    ,dict(mode='photons', chunk_size=5E3, overlap=0., bin_size=5E3,max_chunks=2)|dict(estimator='min-poly', plot=False, output=None)
    ]
new_dir = process_batch(base_in,parse=False,parser_type=parser_type,filter=False,postfilter=False,process=True,bootstrap_dicts=bootstrap_dicts,postprocess=True,max_files=10)

No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)

Available targets: [1000. 5000.]
Chosen target: 1000.0
Chosen target: 5000.0

2.2.3. Closing Remark

These routines are the basic building block to evaluate the data for the other figures of the paper. Depending on the features you are interested in, you may choose different analysis parameters. Have a look at the figure code in ~/src/figures/. Each figure has its own directory that stores data needed to generate the figures and a script evaluate_data.py that allows you to reproduce the data that is plotted in the figure.