Time to allow: ETA ~20 min
This vignette purposes is to show users how to setup their computers for RADseq genomic analysis inside R. It targets users of my packages or workshops.
The vignette as an
Installation problems section, browse through it if your experiencing installation issues.
The vignette is also usefull if your:
grurand it’s dependencies or simulations with
I’m currently merging information also found in this tutorial
Warning: package management software Homebrew and/or MacPorts : My experience with these packages is that at some point they will be unreliable with genomic software installation. It might do the trick for some software, but eventually you will loose a lot of time trying to figure out what’s your problem.
Make sure you have administrator and root user access to your computer how.
Make sure it’s installed…if not, follow instructions. Although the prompt message may be a bit confusing, just click install.
In the finder using keyboard shorcut
cmd+shift+g go to:
/Library/Developer/CommandLineTools/Packages/ and double click on
macOS_SDK_headers_for_macOS_10.14.pkg to install.
xcode-select: error: command line tools are already installed, use "Software Update" to install updates
makein the terminal
./configure: configure everything before installation
make: connect libraries and the source before make install (doc)
make install: use to build (compile) source code to create binary files and install the application on our system as configured using
./configure, usually in
Apple can’t ship GNU Compiler Collection (GCC) with OpenMP enabled, similar story with Clang the other compiler used in macOS. Consequently, both need to be updateed manually if you want to run software that ues parallel computing (like stacks).
The command below fro both GCC and Clang will:
share) of the compiler on your computer
Choose the binary version number based on your OS and change the version accordingly link.
We want clang compiler with OpenMP enabled.
The shell start up script and PATH to programs**
To make things a little easier to talk to your computer, each time you open the Terminal a shell start up scripts tells your computer where to look for programs. The path for your programs can be modified in your shell start up script. When your computer is searching for programs, it looks into these path:
The output should look like this:
/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin. But sometimes, it will also say:
No such file or directory (no worries, see below).
pwd command to know exactly where you are!
The name of the shell startup file differs across platforms. Depending on OS it is called ~/.bash_profile and sometimes ~/.profile. Filename beginning with a dot “.” are reserved for the system and are invisible in the mac Finder.
Find your shell start up script with the following command:
If this returns nothing (blank), you don’t have a shell start up script. Create one with this command
To modify, you can use BBEdit to open or make and modify hidden items (using the option Show hidden items on the open file screen). With Linux, use Vi!
Copy/paste the line below in your
After modifying your shell start up script always run the command
source ~/.bash_profile to reload it.
Below are useful but not essential software you will like to have on your mac.
This software is similar to curl that is already on your Mac, it’s also good to have it. This will help you save a lot of time in the future.
Option 1: trouble-free install
Option 2: from source
Install gnutls dependency
# for novice user: copy one line at a time in the Terminal: cd ~/Downloads curl -O http://www.gnupg.org/ftp/gcrypt/gnutls/v3.5/gnutls-3.5.18.tar.xz tar -zxvf gnutls-3.5.18.tar.xz cd gnutls-3.5.18 ./configure --with-included-libtasn1 --without-p11-kit #Forget about the remaining warnings and proceed with `make` make sudo make install
Now the wget part…
First look for the .tar.gz file with the latest release (currently: wget2-1.99.2.tar.gz) in this folder, Modify (if you need to), Copy & Paste the commands below (one line at a time):
cd ~/Downloads # change the '-x.xx.tar.gz' number below with the latest release version. curl -O http://ftp.gnu.org/gnu/wget/wget2-1.99.2.tar.gz tar -zxvf wget2-1.99.2.tar.gz cd wget2-1.99.2 ./configure make sudo make install # will prompt you to enter your password cd .. # goes up to the enclosing directory wget --help # to test installation!
To remove a folder and downloaded packages use the
sudo rm -R command, but be very careful with this command because the content is deleted FOREVER, it doesn’t go in the trash…
Delete the wget folder and package using
sudo rm -R wget*.
This is TextWrangler replacement and is a free text editor that will help you save time. Once installed, go in the
Apple Menu bar -> BBEdit -> Install Command Line
Make sure you have GCC and CLANG with OpenMP enabled. Several flavors available, check for the proper link
To install R v3.6.3 “Holding the Windsock” released on 2020-02-29 download the installer and follow the instructions
To remove R completely from macOS
To download RStudio, check this link and download the installer for your OS.
Below is how I setup most of my computers after a clean macOS install. 1. Start with devtools and tidyverse
If the console print this:
Do you want to install from sources the package which needs compilation? (Yes/no/cancel). Always aswer no unless of course you know what you are doing.
For some packages you might have to compile from source and the use of different compiler is sometimes very useful. You need to tell R how to use the compilers. This might change from one package to another. Nothing is simple, you know this by now… All this is done through R’s Makevars file located in
To modify or create the file, the fastest way is to use the package usethis (it’s installed automatically with
Makevars content required:
CC=/usr/local/bin/gcc CXX=/usr/local/bin/g++ FC=/usr/local/bin/gfortran F77=/usr/local/bin/gfortran PKG_LIBS = -fopenmp -lgomp PKG_CFLAGS= -O3 -Wall -pipe -pedantic -std=gnu99 -fopenmp PKG_CXXFLAGS=-fopenmp -std=c++11 CFLAGS= -O3 -Wall -pipe -pedantic -std=gnu99 -fopenmp SHLIB_OPENMP_CFLAGS = -fopenmp SHLIB_OPENMP_CXXFLAGS = -fopenmp SHLIB_OPENMP_FCFLAGS = -fopenmp SHLIB_OPENMP_FFLAGS = -fopenmp # change the nex line according to your computer compiler version (use gcc -v in terminal): FLIBS=-L/usr/local/lib/gcc/x86_64-apple-darwin19/9.2.0/finclude CFLAGS=-mtune=native -g -O2 -Wall -pedantic -Wconversion CXXFLAGS=-mtune=native -g -O2 -Wall -pedantic -Wconversion
Sometimes you’ll get warnings while installing dependencies required for x package.
#Warning: cannot remove prior installation of package ‘stringi’
To solve this problem, delete manually the problematic package in the installation folder (on mac:
/Library/Frameworks/R.framework/Resources/library) or in the
Using the latest version of R, RStudio and packages is recommended. If your heart start pounding just at the thought of having to install a new R version, you should have a look at packrat.
Look at the output in R console when you get an error message. If it’s related to one’s of the packages dependencies, try installing it separately before attempting to reinstall the problematic package.
Required if GCC compiler is used (TLS backend is then used). Not required if clang is used (securetransport backend is used).
#In browser https://www.openssl.org/source/openssl-1.1.1.tar.gz #In Terminal cd Downloads curl -O https://www.openssl.org/source/openssl-1.1.1.tar.gz tar -zxvf openssl-1.1.1.tar.gz cd openssl-1.1.1 ./config make -j12 #change with your number of CPU make test #long sudo make install cd .. sudo rm -R openssl*
Check for the latest release of curl
The next step depends on the compulier used
Note: with macOS 10.15 this give me an error
configure: error: OpenSSL libs and/or directories were not found where specified!
If this is the case, use clang:
Check that you have a file called
If you dont have the file:
Add this to your
.Renviron file located in
Use BBEdit or other software that allows you to see hidden files starting with a dot
math.hor any other
An example during
clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/progress/include" -I/usr/local/include -Imio/include -DWIN32_LEAN_AND_MEAN -Ispdlog/include -fPIC -Wall -g -O2 -c gen.cc -o gen.o clang++ -std=gnu++11 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/progress/include" -I/usr/local/include -Imio/include -DWIN32_LEAN_AND_MEAN -Ispdlog/include -fPIC -Wall -g -O2 -c index_collection.cc -o index_collection.o /usr/local/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/3.5/Resources/library/progress/include" -I/usr/local/include -fPIC -I/usr/local/include -c localtime.c -o localtime.o localtime.c:42:10: fatal error: 'string.h' file not found #include <string.h> ^~~~~~~~~~
Its a problem with a previous version of
pbmcapply and it’s interaction with
RStudio > Session > Restart R)
Note: if your heart start pounding just at the thought of having to update everything on your computer you should definitely have a look at packrat: it’s very easy to use.
If you have a PC and you’re getting this error or closely related error:
parallel.core = 1 in the function generating the error.
Here is an example below:
Error in dyn.load(file, DLLpath = DLLpath, ...) : unable to load shared object '/home/rstudio/R/x86_64-pc-linux-gnu-library/3.6/units/libs/units.so': libudunits2.so.0: cannot open shared object file: No such file or directory Calls: <Anonymous> ... asNamespace -> loadNamespace -> library.dynam -> dyn.load Execution halted
Sys.getenv("LD_LIBRARY_PATH") # ""
Note that if the output is not empty, like in the example above, write down the output.
/usr/local/lib/to the output above:
When it’s empty:
# in R: Sys.setenv(LD_LIBRARY_PATH="/usr/local/lib/") # For Linux you could use: /usr/local/lib/:/usr/lib64
When it’s not, add at the end, separated by
So far, I haven’t found the cure to this computer-specific problem.
Download the latest github release of Allan Strand’s rmetasim
If you want to use more loci during your simulations (default is 10001), you need to modify rmetasim before compiling. With a text editor, modify the
const.h file in the
rmetasim-master/src/const.h. Navigate to lane 33 and change the integer to the desired maximum number of loci. Or do this in the Terminal:
rmetasim requires these Makevars (
~/.R/Makevars file) specifications. If you have other lines, comment (#) before compiling
With most OS (including macOS 10.14):
CC=/usr/local/bin/clang CC=/usr/local/bin/clang CXX=/usr/local/bin/clang++ CXX1X=/usr/local/bin/clang++ CXX11=/usr/local/bin/clang++ FLIBS=-L/usr/local/lib CFLAGS=-I/usr/local/include LDFLAGS=-L/usr/local/lib CPPFLAGS=-I/usr/local/include SHLIB_OPENMP_CFLAGS= -fopenmp SHLIB_OPENMP_FCFLAGS= -fopenmp SHLIB_OPENMP_FFLAGS= -fopenmp SHLIB_OPENMP_CXXFLAGS= -fopenmp
This latest os requires extra lines:
CC=/usr/local/bin/clang CC=/usr/local/bin/clang CXX=/usr/local/bin/clang++ CXX1X=/usr/local/bin/clang++ FLIBS=-L/usr/local/lib LDFLAGS=-L/usr/local/lib SHLIB_OPENMP_CFLAGS= -fopenmp SHLIB_OPENMP_FCFLAGS= -fopenmp SHLIB_OPENMP_FFLAGS= -fopenmp SHLIB_OPENMP_CXXFLAGS= -fopenmp CFLAGS+=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk CCFLAGS+=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk CXXFLAGS+=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk CPPFLAGS+=-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk
If you’re getting this error:
"https" not supported or disabled in libcurl, extra steps are required, check the installation problems section to install
curl and enabling
Modify in a text editor the
configure file inside the
XGBoost requires these Makevars specifications. If you have other lines, comment (#) before compiling:
CC=/usr/local/bin/gcc CXX=/usr/local/bin/g++ CXX11=/usr/local/bin/g++ CXX14=/usr/local/bin/g++ CXX17=/usr/local/bin/g++ SHLIB_OPENMP_CFLAGS= -fopenmp SHLIB_OPENMP_FCFLAGS= -fopenmp SHLIB_OPENMP_FFLAGS= -fopenmp SHLIB_OPENMP_CXXFLAGS= -fopenmp CFLAGS=-g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe CXXFLAGS=-g -O3 -Wall -pedantic -std=c++11 -mtune=native -pipe LDFLAGS=-L/usr/local/lib -Wl,-rpath,/usr/local/lib CPPFLAGS=-I/usr/local/include -I/usr/local/include
You should see a time difference between both runs
LightGBM requires an OpenMP-enabled compiler. Currently, it doesn’t work well with clang, so make sure you have updated your GCC compiler (instructions above). Additionally,
LightGBM requires CMake
To add CMake to the PATH:
PATH="/Applications/CMake.app/Contents/bin":"$PATH" # Or, to install symlinks to '/usr/local/bin', run: sudo "/Applications/CMake.app/Contents/bin/cmake-gui" --install # Or, to install symlinks to another directory, run: sudo "/Applications/CMake.app/Contents/bin/cmake-gui" --install=/path/to/bin #Then, run the following commands to install LightGBM:
randomForestSRC requires the GCC OpenMP-enabled compiler to run in parallel. See instructions above if not already done.
Check that the lines below are not commented in your
CC=/usr/local/bin/gcc CXX=/usr/local/bin/g++ CFLAGS=-g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe CXXFLAGS=-g -O3 -Wall -pedantic -std=c++11 -mtune=native -pipe PKG_CFLAGS= -O3 -Wall -pipe -pedantic -std=gnu99 -fopenmp PKG_CXXFLAGS=-fopenmp -std=c++11 FC=/usr/local/bin/gfortran F77=/usr/local/bin/gfortran LDFLAGS=-L/usr/local/lib PKG_LIBS = "-liconv"
From the Terminal run these steps to download and compile randomForestSRC:
Make sure you have autoconf installed:
autoconf: error: no input file, if not, install following the steps here.
You want to make sure that this line is printed during execution of the previous command:
checking whether OpenMP will work in a package... yes or
checking for /usr/local/bin/gcc option to support OpenMP... -fopenmp
Re-start R and the software fastsimcoal2 will automatically be in your PATH…
To install COLONY 30/07/2018, V126.96.36.199:
The old openmpi version (openmpi-1.6.5) is required, saddly.
cd ~/Downloads curl -O https://download.open-mpi.org/release/open-mpi/v1.6/openmpi-1.6.5.tar.gz tar -zxvf openmpi-1.6.5.tar.gz cd openmpi-1.6.5 export TMPDIR=/tmp ./configure F77=gfortran #--prefix=/usr/local -openmp # no longer work for some reason make -j 12 sudo make install sudo rm -R ~/Downloads/openmpi*
To download COLONY, follow instructions on Jinliang Wang ZSL website. The file you need to uncompress is named:
To download COLONY, follow instructions on Jinliang Wang ZSL website. The file you need to uncompress is named:
Several options are available depending on the compiler you have installed.
macOS Catalina comes with Github, a Version Control System (VCS), pre-installed. However, the install is in
/usr/bin/git which can make it difficult for beginners to update. To change this, run these commands:
git --version # show current git version installed which git # returns where is git on your computer cd ~/Downloads git clone https://github.com/git/git # install the latest Git cd git make configure ./configure make -j12 sudo make install cd .. sudo rm -R ~/Downloads/git/ # remove git folder source ~/.bash_profile # reload startup script git --version # confirmed the version you just installed which git # returns /usr/local/bin
In System Preferences choose
Keyboard -> Shortcuts. From the left panel, choose Services. In the right panel, under Files and Folders, choose New Terminal at Folder and/or New Terminal Tab at Folder. Now you can right-click your track pad or mouse on a folder and choose Services -> New Terminal at Folder!
Open the Automator application (it’s on all MAC…).
Go in the Finder, select a folder and right click on it you should see ‘copy path to clipboard’ at the bottom or in Services.