GNU Astronomy Utilities

Gnuastro 0.19 News: Warp (align FITS image by WCS) and ConvertType (annotations from catalogs).

GNU Astronomy Utilities — Sun, 09 Oct 2022 17:08:34 GMT

GNU Astronomy Utilities (Gnuastro) is a large collection of command-line programs and C/C++ libraries for astronomical data analysis. In this vide, we introduce two of the newly added features in version 0.19: WCS alignment of images (while correcting any projections, distortions or rotations) and adding vector graphics annotations from a catalog.

SMACK 13: Introduction to Containers (Docker and Singularity)

GNU Astronomy Utilities — Mon, 05 Sep 2022 12:58:39 GMT

Containers are portable environments that package up code and all its dependencies so that an application can run quickly and reliably from one computing environment to another. Most people are probably familiar with full virtualization environments (such as VirtualBox), so in this talk we will explain the main differences between full virtualization and containers (sometimes called light-weight virtualization), and when to use each.

At the same time, not all container technologies have the same goals and/or approaches. Docker is the most mature container offering, but it is geared mainly towards micro-services. Singularity is a newer contender, with an emphasis on mobility of compute for scientific computing. We will introduce both softwares, showing how to create and use containers with each of them, while discussing real-life examples of their use.

The lecture notes can be found here: https://gitlab.com/makhlaghi/smack-talks-iac/-/blob/master/smack-13-docker.md

SMACK 17: Basic LaTeX (hands-on) 2

GNU Astronomy Utilities — Mon, 05 Sep 2022 11:54:35 GMT

In the 16th SMACK, the very basics of LaTeX were introduced; showing how to make a basic document from scratch, set the printable size, inserting images and tables and etc. In this session, we will go into more advanced features that are also commonly helpful when preparing a professional document (while letting you focus on your exciting scientific discovery, and not have to worry about the style of the output). These features include automatically referencing different parts of your document using labels (this allows you to easily shuffle figures, sections or tables), making all references to various parts of your text click-able (greatly simplifying things for your readers), using Macros (to avoid repetition or importing your analysis results automatically), adding bibliography, keeping your LaTeX source, and your top directory clean, and finally using Make to easily automate the production of your document.

Lecture notes: https://gitlab.com/makhlaghi/smack-talks-iac/-/blob/master/smack-17-latex-b.md

SMACK 16: Basic LaTeX (hands-on) 1

GNU Astronomy Utilities — Mon, 05 Sep 2022 11:45:13 GMT

LaTeX is a professional typesetting system to create a ready-to-print or publish (usually PDF!) document (usually papers!). LaTeX is the format used by arXiv, and many journals, when you want to submit your scientific papers. With LaTeX, you "program" the final document: text, figures, tables, bibliography and etc, through a plain-text (source) file. When you run LaTeX on your source, it will parse your LaTeX source and find the best way to blend everything in a nice and professionally design PDF document. Therefore LaTeX allows you to focus on the actual content of your text, tables, plots, and not have to worry about the final design (the "style" file provided by the journal will do all of this for you automatically). This is in contrast to "What-You-See-Is-What-You-Get" (or WYSIWYG) editors, like Microsoft Word or LibreOffice Writer, which force you to worry about style in the middle of content writing (which is very frustrating). Since the source of a LaTeX document is plain-text, you can use version-control systems like Git to keep track of your changes and updates (Git was introduced in SMACK 5, SMACK 6 and SMACK 8). This feature of LaTeX allows easy collaboration with your co-authors, and is the basis of systems like Overleaf. Previously (in SMACK 11), some advanced tips and tricks were given on the usage of LaTeX. This SMACK session is aimed to complement that with a hands-on introduction for those who are just starting to use LaTeX. It is followed by SMACK 17 on other basic features that are necessary to get comfortable with LaTeX.

Lecture notes: https://gitlab.com/makhlaghi/smack-talks-iac/-/blob/master/smack-16-latex.md

SMACK 14: Make: a powerful tool to manage your projects

GNU Astronomy Utilities — Tue, 16 Aug 2022 18:06:34 GMT

Did you ever want to re-run your project from the beginning, but run into trouble because you forgot one step? Do you want to run just one part of your project and ignore the rest? Do you want to run it in parallel with many different inputs using all the cores of your computer? Do you want to design a modular project, with re-usable parts, avoiding long files hard to debug?

Make is a well-tested solution to all these problems. It is independent of the programming language you use. Instead of having a long code hard to debug, you can connect its components making a chain. Make will allow you to automate your project and retain control of how its parts are integrated. This SMACK seminar will give an overview of this powerful tool.

Maneage: Managing data lineage for long-term and archivable reproducibility (Invited talk at ESO's ROSA2022)

GNU Astronomy Utilities — Sun, 15 May 2022 14:29:16 GMT

The increasing volume, diversity, and role of data in modern research has been very fruitful. However, these same factors, have also made it harder to describe (in sufficient detail) the processing behind a scientific result within the confines of a traditional paper. It is thus becoming harder and harder to reproduce results (i.e., critically review by coauthors, referees or larger community) that define scientific progress. In this talk, Maneage (MANaging data linEAGE) is introduced as a working solution to this problem. Maneage is a template that should be customized for every project. It will enable exact reproduce a scientific analysis (from the input data and software, to the processing and creation of final report, paper or dataset. The necessary software are built (from the low-level C compiler and shell, to the higher-level science programs and all their dependencies) with the predefined configuration. The data are imported from pre-determined URLs (and validated with recorded checksums). The software are then run on the input data sets to produce the final result. The template will finally produce a "dynamic" PDF using LaTeX macros: any change in the analysis will automatically update the relevant parts of the PDF (for example numbers, tables or figures). A project defined in this template is fully managed and published in plain text and only consumes a few hundred kilo-bytes (unlike binary multi-gigabyte blobs like containers). It is thus easily to publish, for example on arXiv with the paper's LaTeX source or Software Heritage. Giving readers the ability to exactly study and reproduce the paper's results, for the long term. It is also easily search-able, providing a treasure trove to extract metadata on the project (even after publication, and without the author's active involvement). This can be very valuable when implement widely (e.g., using machine learning on many project sources to define automatic workflows). But most importantly it will allow other scientists to independently study the details, verify in practice, and build incrementally on each others' work, without necessarily needing to run it.

Maneage webpage: http://maneage.org
Maneage paper: https://arxiv.org/abs/2006.03018
ROSA2022 webpage: https://www.eso.org/sci/meetings/2022/REPRODUCIBILITY2022/programme.html

Gnuastro tutorial at ADASS (Astronomical Data Analysis Software & Systems) 2021

GNU Astronomy Utilities — Sat, 16 Apr 2022 23:22:49 GMT

Mohammad Akhlaghi,Raul Infante-Sainz,Zahra Sharbaf

Gnuastro is an official GNU package of a large collection of programs to enable easy, robust, and most importantly fast and efficient, data analysis directly on the command-line. For example it can perform arithmetic operations on image pixels or table columns/rows, visualize FITS images as JPG or PDF, convolve an image with a given kernel or matching of kernels, perform cosmological calculations, crop parts of large images (possibly in multiple files), manipulate FITS extensions and keywords, and perform statistical operations. In addition, it contains
programs to make catalogs from detection maps, add noise, make mock profiles with a variety of radial functions using monte-carlo integration for their centers, match catalogs, and detect objects in an image among many other operations. Gnuastro is written to comply fully with the GNU coding standards and integrates well with all Unix-like operating systems. This enables astronomers to expect a fully familiar experience in the building, installing and command-line user interaction that they have seen in all the other GNU software that they use (core components in many Unix-like operating systems). Gnuastro's extensive library is also installed for users who want to build their own unique/custom programs.

NoiseChisel, Segment and Gnuastro: non-parametric detection, segmentation and catalog production

GNU Astronomy Utilities — Wed, 02 Mar 2022 13:18:27 GMT

Introduction to NoiseChisel, presented at Institut d'Astrophysique de Paris (IAP) on December 2nd, 2021.

SMACK 4 Shell scripting for automation and text editors

GNU Astronomy Utilities — Fri, 30 Jul 2021 16:47:27 GMT

In the previous SMACKs, basic operations on the command-line were reviewed in interactive mode: where you enter one command, wait for it to finish and enter the next command. The shell's interactive mode is good for small/fast tests, but is not scalable. For example when you know the commands and parameters work and want to apply them to hundreds/thousands of targets. Shell scripts are simply plain-text files that store all the various commands that you want to be executed in sequence. They are designed precisely for the easier management of operations that are more complex than a simple command. In fact many of the commands in the Unix-like operating systems are actually shell scripts! Here will review some basic intricacies with designing robust shell scripts and avoiding common pitfalls. Also, since shell scripts are simple plain-text files, we will also do a short review of simple plain-text editors like GNU Nano and more advanced editors like Vim and GNU Emacs. Advanced editors have many useful features to simplify programming in many languages (including shell programming) and don't need complex graphic environments and can be run in the raw command-line (a major advantage when scaling your project up to supercomputers that don't have a graphic environment). For newcomers to data-intensive astronomical research, we encourage you to select an advanced editor, and master it early to greatly simply your research (in any language).

SMACK 3 GNU Astronomy Utilities (Gnuastro) hands on

GNU Astronomy Utilities — Fri, 30 Jul 2021 16:33:32 GMT

Gnuastro is an official GNU package that is currently maintained at the IAC. It is a large collection of programs to enable easy, robust, fast and efficient data analysis directly on the command-line. For example it can perform arithmetic operations on image pixels or table columns/rows without having to write programs, visualize FITS images as JPG or PDF, convolve an image with a given kernel or matching of kernels, perform cosmological calculations, crop parts of large images (possibly in multiple files), manipulate FITS extensions and keywords, and perform statistical operations. In addition, it contains programs to make catalogs from detection maps, add noise, make mock profiles with a variety of radial functions using monte-carlo integration for their centers, match catalogs, and detect objects in an image among many other operations. Gnuastro is written to comply fully with the GNU coding standards and integrates well with all Unix-like operating systems. This enables astronomers to expect a fully familiar experience in the building, installing and command-line user interaction that they have seen in all the other GNU software that they use (core components in many Unix-like/POSIX operating systems). In this SMACK, we will introduce Gnuastro and demonstrate a selection of its commonly used features. Relevant links are as follows:

Lecture-notes: in https://gitlab.com/makhlaghi/smack-talks-iac/-/blob/master/smack-3-gnuastro.md
Gnuastro's main webpage: https://www.gnu.org/s/gnuastro
Gnuastro documentation: https://www.gnu.org/s/gnuastro/manual
Gnuastro tutorials: https://www.gnu.org/s/gnuastro/manual/html_node/Tutorials.html

SMACK 2 POSIX power tools Command-line tools and shell scripting

GNU Astronomy Utilities — Fri, 30 Jul 2021 16:17:49 GMT

The shell (or command-line) is the most commonly used (for advanced operations) interface to Unix-like operating systems. In this session we'll introduce some of the most commonly used command-line/shell features and how they may be used in a hypothetical research project, using real astronomical data sets. The session plan (with a listing of used commands and short explanation) is available here: https://gitlab.com/makhlaghi/smack-talks-iac/-/blob/master/smack-2-shell.md

SMACK 1 History of the POSIX operating systems and basic linux command-line

GNU Astronomy Utilities — Fri, 30 Jul 2021 16:03:49 GMT

An introduction to the Unix-like operating systems, starting with a review of the very interesting history behind it.

NoiseChisel: how to detect diffuse signal in noise (hands-on)

GNU Astronomy Utilities — Fri, 30 Jul 2021 14:47:37 GMT

Hands-on introduction on the inner working of NoiseChisel by Dr. Mohammad Akhlaghi. NoiseChisel is a program in GNU Astronomy Utilities (Gnuastro) that is in charge of separating signal from noise in a 2D image or 3D datacube (and as a side-effect, measuring an accurate background level for the data). This video was made with Gnuastro 0.15 (released on 2021-05-31). Updates in NoiseChisel will be published in the Gnuastro manual (discussed in video). See the links below for more related video, text and software:

Video of colloquium on NoiseChisel: http://a.impartus.com/ilc/w/v/VQoO
Video of hands-on Gnuastro demo: https://www.youtube.com/watch?v=6mp5-...
Slides of NoiseChisel talk: http://akhlaghi.org/pdf/noisechisel.pdf
First (main) NoiseChisel paper: https://arxiv.org/abs/1505.01664
Paper on improvements after 10 versions: https://arxiv.org/abs/1909.11230
Gnuastro webpage: http://www.gnu.org/software/gnuastro/
Gnuastro in Conda: https://anaconda.org/conda-forge/gnua...
Gnuastro manual (in many formats): http://www.gnu.org/software/gnuastro/...
Gnuastro manual (PDF): http://www.gnu.org/software/gnuastro/...
Tutorials to get started: https://www.gnu.org/software/gnuastro...
Mohammad Akhlaghi (presenter): http://akhlaghi.org