Internal:DQHED

Distributed Quantum Hadamard Edge Detection (DQHED)

Literature Review

Sections 1 and 2 of this report was generated by reviewing publicly available research as of March 2022.

Methods

The methods used can be found in Section 3.1 of this report.

Weekly Reports

Weeks of 2Mar and 12Mar

Generated the following notes on current methods in quantum edge detection.

Week of 19Mar

In an effort to understand it - reviewed, commented, and added typing to Emmanuel's QHED code in the quantum repository at wahab.hpc.odu.edu:/RC/group/RC_NIKOS/CodeRepository/quantum.git

Weeks of 26Mar and 2Apr

Created a separate script for Multi-node Quantum Hadamard Edge Detection (QHED) with a workflow for parallel quantum circuit simulation and execution. Current progress is stored on the qhed branch. The three main tasks eligible for concurrency are:

1. Circuit description

2. Transpilation

3. Execution

In addition, the requirement for padding was removed, as it is not necessary. Only one pixel per run is wasted to provide overlap between subimages.

Week of 9Apr

Researched the SLURMCluster module from Dask with respect to the MQHED problem. All that is required is to provide the module a description of the nodes. The Scheduler will create and submit a job script to SLURM, which will function to automatically request nodes. Once this is complete, Dask Futures can be used to provide arbitrary Python methods with arguments for the SLURMCluster to manage. Relevant methods are dask.distributed.Client.map and dask.distributed.[result, gather, as_completed]. Sample usage will be as follows, with the SLURMCluster parameters specified by a .yaml file:

   from dask.distributed import Client, as_completed
   from dask.jobqueue import SLURMCluster

   cluster = SLURMCluster()
   cluser.adapt(minimum_jobs=min, maximum_jobs=max)
   client = Client(cluster)
   .
   .
   .
   results = []
   futures = client.map(func, args)
   results = client.gather(futures)    

The quantum portion of the algorithm has been implemented and run on an 11th Gen Intel Core i7-11800H (16 physical cores, 32 logical cores). It remains to implement the code process the results of the circuit runs and convert them into an edge detected image.

Week of 16Apr

The code example from the 9Apr section fails to allocate any cores or memory on Wahab, for reasons unknown. An alternative method of using a batch script with srun and possibly mpirun to execute a Python script which calls dask_mpi.initialize() and Client() has been successful on a sample script, but it is extremely slow and essentially infeasible for large numbers of mpi slots. An example is provided below

   from dask.distributed import Client, progress
   import dask_mpi

   dask_mpi.initialize(interface='ib0') # Optional parameter to specify InifiniBand
   client = Client()
   .
   .
   .
   futures = client.map(func, args)
   progress(futures)

HPC ITS has been contacted and have suggested that using dask_mpi with srun, but not mpirun, is the best course of action. However, this method is currently much slower than running dask on a local machine with the LocalCluster provided to the Client object. It has demonstrated itself to be infeasible in the current configuration when running a simple program with more than 4 allocated tasks.

Further experimentation in using dask with the Wahab SLURM scheduler is needed to determine its efficacy. It may be advisable to get in contact with dask contributors to help implement dask on ODU CS HPC clusters, as it would benefit both our group and the school. If it deemed infeasible, other methods or software may need to be investigated.

IT has corrected some of the errors, but the dask_mpi.initialize() method still seems to take a prohibitive amount of time to set up communication between processors/nodes. Further investigation is required.

Week of 23Apr

A fix was found for the dask-jobqueue cluster module. The issue was the module was not prepending the required environment flags to the batch scripts it generated. A workaround was found by using the `env_extra` parameter in the SLURMCluster constructor to prepend module loading and the `crun` command to the script. There are a number of other useful options in Dask which might be helpful in the future. It should also be noted that dask-jobqueue does not just create a batch script. It creates a batch script which forces the creation of a dask.worker with the following python module command:

   python -m distributed.cli.dask_worker

With this fix, a text script was successfully run which allocated the correct number of jobs across the correct number of nodes. It now remains to refactor the DQHED script for the multi-node environment, as my new understanding of the IPC subtleties makes me want to refactor a few parts of the existing script.

The current, unoptimized code was run on a 64 x 64 image and 256 x 256 image using 5 qubits. The results can be seen in the table below. Strange errors were encountered when trying to run the 256 x 256 image with more cores, so that will need to be addressed.

Week of 30Apr

Finished results extraction and image reconstruction implementations. This, in turn, revealed that the current quantum circuit implementation is just an image encoder, not edge detection. The circuit must be modified to encode redundant pixels and then add a Hadamard gate. From top to bottom, the images in this section show the image with no thresholding, low thresholding, and high thresholding. The original image resolution is 1024 x 1024.

Subimage Extraction

Qiskit returns a dictionary of circuit results, of which the keys are the strings representing binary numbers (i.e. "001001"). This must be converted back into an array representing the subimage from which the quantum circuit was built. Each even index is removed, leaving the relevant differences.

Image Reconstruction

After extraction, all subimages must be aggregated to reconstruct the edge-detected image. This is difficult for Python, since the GIL prevents the use of shared memory. Thus, each subimage must be saved to the final data structure sequentially.

Original 1028 x 1028 image

256x256 pixel encoded image

256x256 pixels, high threshold

128x128 pixels, low threshold

Week 0f 7May

The edge detection circuit was fixed and the results of running the script on an 11th Gen Intel Core i7-11800H (16 physical cores, 32 logical cores) can be seen below:

128 x 128 edge detection, FakeHanoi, low threshold

256 x 256 edge detection, FakeHanoi, low threshold

256 x 256 edge detection, FakeHanoi, moderate threshold

The script will now be refactored to decrease Inter-Process Communication (IPC) and memory loading.

After refactoring, the script was able to handle 1028x1028 images. The algorithm is clearly highly-susceptible to noise, as each pixel only has knowledge of its neighboring pixels. The absoluteness of the algorithm and thresholding also make finding brighter edges more likely, as can be seen with the clarity of the skull in the edge detected image. The fact that it does highlight the tissue border surrounding the skull does prove that the algorithm is, if fact detecting edges. However, that edge is much less defined that that of the skull.

1028 x 1028 edge detection, FakeHanoi, low threshold

1028 x 1028 edge detection, FakeHanoi, moderate threshold

Week of 14May

When running on the SLURM cluster, the worker processes are having trouble serializing the user-defined imported modules. This will be the problem to solve this week. Otherwise, there should hopefully be no issues.

After discussing with Emmanuel, the decision was made to use the Aer Simulator instead of any noisy simulator, to achieve ideal results first. The following images are the incorporation of the 2-axis QHED algorithm using the Qiskit Aer Simulator

Results of DQHED on 1024x1024 image, AerSimulator, high threshold

Results of DQHED on 1024x1024 image, AerSimulator, moderate threshold

Results of DQHED on 1024x1024 image, AerSimulator, low threshold

Results of DQHED on 1024x1024 image, AerSimulator, minimal threshold

Week of 21May

Ran testing on the optimized two-dimensional DQHED script. The run times are provided below, for both the full-usage configuration (above) and the DASK recommended node usage configuration (below).

DQHED2D results for full core usage

DQHED2D results for recommended core usage

Week of 28May (Current Week)

DQHED3D results

References

Emmanuel's Contibutions

Proposal Additions for QHED December 2022[1]

Software

Qiskit

Dask