Advanced Computing Platform for Theoretical Physics

commit大文件会使得服务器变得不稳定,请大家尽量只commit代码,不要commit大的文件。

Commit 17e44adb authored by chenzhiyuan's avatar chenzhiyuan
Browse files

Initial commit

parents
Pipeline #1156 failed with stages
.*
!.gitignore
*.o
*.exe
*.swp
lib
packages
build
bin/hipInfo
bin/hipBusBandwidth
bin/hipDispatchLatency
bin/hipify-clang
include/hip/hcc_detail/hip_prof_str.h
include/hip/hip_version.h
tags
samples/1_Utils/hipInfo/hipInfo
samples/1_Utils/hipBusBandwidth/hipBusBandwidth
samples/1_Utils/hipDispatchLatency/hipDispatchLatency
\ No newline at end of file
This diff is collapsed.
# Contributor Guidelines
## Make Tips
When building HIP, you will likely want to build and install to a local user-accessible directory (rather than /opt/rocm).
This can be easily be done by setting the -DCMAKE_INSTALL_PREFIX variable when running cmake. Typical use case is to
set CMAKE_INSTALL_PREFIX to your HIP git root, and then ensure HIP_PATH points to this directory. For example
```
cmake .. -DCMAKE_INSTALL_PREFIX=..
make install
export HIP_PATH=
```
After making HIP, don't forget the "make install" step !
## Adding a new HIP API
- Add a translation to the hipify-clang tool ; many examples abound.
- For stat tracking purposes, place the API into an appropriate stat category ("dev", "mem", "stream", etc).
- Add a inlined NVCC implementation for the function in include/hip/nvcc_detail/hip_runtime_api.h.
- These are typically headers
- Add an HCC definition and Doxygen comments for the function in include/hcc_detail/hip_runtime_api.h
- Source implementation typically go in src/hcc_detail/hip_hcc.cpp. The implementation may involve
calls to HCC runtime or HSA runtime, or interact with other pieces of the HIP runtime (ie for
hipStream_t).
#### Testing HCC version
In some cases new HIP features are tied to specified releases of HCC, and it can be useful to determine at compile-time
if the current HCC compiler is sufficiently new enough to support the desired feature. The `__hcc_workweek__` compiler
define is a monotonically increasing integer value that combines the year + workweek + day-of-week (0-6, Sunday is 0)
(ie 15403, 16014, etc).
The granularity is one day, so __hcc_workweek__ can only be used to distinguish compiler builds that are at least one day apart.
```
#ifdef __hcc_workweek_ > 16014
// use cool new HCC feature here
#endif
```
Additionally, hcc binary can print the work-week to stdout: ("16014" in the version info below.)4
```
> /opt/rocm/hcc/bin/hcc -v
HCC clang version 3.5.0 (based on HCC 0.8.16014-81f8a3f-f155163-5a1009a LLVM 3.5.0svn)
Target: x86_64-unknown-linux-gnu
Thread model: posix
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.8
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.8.4
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.9
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.9.1
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.8
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Candidate multilib: x32;@mx32
Selected multilib: .;@m64
```
The unix `date` command can print the HCC-format work-week for a specific date , ie:
```
> date --utc +%y%U%w -d 2015-11-09
15451
```
## Unit Testing Environment
HIP includes unit tests in the tests/src directory.
When adding a new HIP feature, add a new unit test as well.
See [tests/README.md](README.md) for more information.
## Development Flow
It is recommended that developers set the flag HIP_BUILD_LOCAL=1 so that the unit testing environment automatically rebuilds libhip_hcc.a and the tests when a change it made to the HIP source.
Directed tests provide a great place to develop new features alongside the associated test.
For applications and benchmarks outside the directed test environment, developments should use a two-step development flow:
- #1. Compile, link, and install HCC. See [Installation](README.md#Installation) notes.
- #2. Relink the target application to include changes in the libhip_hcc.a file.
## Environment Variables
- **HIP_PATH** : Location of HIP include, src, bin, lib directories.
- **HCC_HOME** : Path to HCC compiler. Default /opt/rocm/hcc.
- **HSA_PATH** : Path to HSA include, lib. Default /opt/rocm/hsa.
- **CUDA_PATH* : On nvcc system, this points to root of CUDA installation.
### Contribution guidelines ###
Features (ie functions, classes, types) defined in hip*.h should resemble CUDA APIs.
The HIP interface is designed to be very familiar for CUDA programmers.
Differences or limitations of HIP APIs as compared to CUDA APIs should be clearly documented and described.
## Coding Guidelines (in brief)
- Code Indentation:
- Tabs should be expanded to spaces.
- Use 4 spaces indentation.
- Capitalization and Naming
- Prefer camelCase for HIP interfaces and internal symbols. Note HCC uses _ for separator.
This guideline is not yet consistently followed in HIP code - eventual compliance is aspirational.
- Member variables should begin with a leading "_". This allows them to be easily distinguished from other variables or functions.
- {} placement
- For functions, the opening { should be placed on a new line.
- For if/else blocks, the opening { is placed on same line as the if/else. Use a space to separate {/" from if/else. Example
'''
if (foo) {
doFoo()
} else {
doFooElse();
}
'''
- namespace should be on same line as { and separated by a space.
- Single-line if statement should still use {/} pair (even though C++ does not require).
- Miscellaneous
- All references in function parameter lists should be const.
- "ihip" = internal hip structures. These should not be exposed through the HIP API.
- Keyword TODO refers to a note that should be addressed in long-term. Could be style issue, software architecture, or known bugs.
- FIXME refers to a short-term bug that needs to be addressed.
- HIP_INIT_API() should be placed at the start of each top-level HIP API. This function will make sure the HIP runtime is initialized,
and also constructs an appropriate API string for tracing and CodeXL marker tracing. The arguments to HIP_INIT_API should match
those of the parent function.
- ihipLogStatus should only be called from top-level HIP APIs,and should be called to log and return the error code. The error code
is used by the GetLastError and PeekLastError functions - if a HIP API simply returns, then the error will not be logged correctly.
- All HIP environment variables should begin with the keyword HIP_
Environment variables should be long enough to describe their purpose but short enough so they can be remembered - perhaps 10-20 characters, with 3-4 parts separated by underscores.
To see the list of current environment variables, along with their values, set HIP_PRINT_ENV and run any hip applications on ROCm platform .
HIPCC or other tools may support additional environment variables which should follow the above convention.
#### Presubmit Testing:
Before checking in or submitting a pull request, run all directed tests (see tests/README.md) and all Rodinia tests.
Ensure pass results match starting point:
```shell
> cd examples/
> ./run_all.sh
```
#### Checkin messages
Follow existing best practice for writing a good Git commit message. Some tips:
http://chris.beams.io/posts/git-commit/
https://robots.thoughtbot.com/5-useful-tips-for-a-better-commit-message
In particular :
- Use imperative voice, ie "Fix this bug", "Refactor the XYZ routine", "Update the doc".
Not : "Fixing the bug", "Fixed the bug", "Bug fix", etc.
- Subject should summarize the commit. Do not end subject with a period. Use a blank line
after the subject.
## Doxygen Editing Guidelines
- bugs should be marked with @bugs near the code where the bug might be fixed. The @bug message will appear in the API description and also in the
doxygen bug list.
## Other Tips:
### Markdown Editing
Recommended to use an offline Markdown viewer to review documentation, such as Markdown Preview Plus extension in Chrome browser, or Remarkable.
## Table of Contents
<!-- toc -->
- [Installing pre-built packages](#installing-pre-built-packages)
* [Prerequisites](#prerequisites)
* [HIP-hcc](#hip-hcc)
* [HIP-clang](#hip-clang)
* [HIP-nvcc](#hip-nvcc)
* [Verify your installation](#verify-your-installation)
- [Building HIP from source](#building-hip-from-source)
* [HCC Options](#hcc-options)
+ [Using HIP with the AMD Native-GCN compiler.](#using-hip-with-the-amd-native-gcn-compiler)
<!-- tocstop -->
# Installing pre-built packages
HIP can be easily installed using pre-built binary packages using the package manager for your platform.
## Prerequisites
HIP code can be developed either on AMD ROCm platform using hcc or clang compiler, or a CUDA platform with nvcc installed:
## HIP-hcc
* Add the ROCm package server to your system as per the OS-specific guide available [here](https://rocm.github.io/ROCmInstall.html#installing-from-amd-rocm-repositories).
* Install the "hip-hcc" package. This will install HCC and the HIP porting layer.
```
apt-get install hip-hcc
```
* Default paths and environment variables:
* By default HIP looks for hcc in /opt/rocm/hcc (can be overridden by setting HCC_HOME environment variable)
* By default HIP looks for HSA in /opt/rocm/hsa (can be overridden by setting HSA_PATH environment variable)
* By default HIP is installed into /opt/rocm/hip (can be overridden by setting HIP_PATH environment variable).
* Optionally, consider adding /opt/rocm/bin to your PATH to make it easier to use the tools.
## HIP-clang
* Using clang to compile HIP program for AMD GPU is under development. Users need to build LLVM, clang, lld, ROCm device library, and HIP from source.
* Install the [rocm](http://gpuopen.com/getting-started-with-boltzmann-components-platforms-installation/) packages. ROCm will install some of the necessary components, including the kernel driver, HSA runtime, etc.
* Build HIP-Clang
```
git clone https://github.com/llvm/llvm-project.git
mkdir -p build && cd build
cmake -DCMAKE_INSTALL_PREFIX=/opt/rocm/llvm -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=1 -DLLVM_TARGETS_TO_BUILD="AMDGPU;X86" -DLLVM_EXTERNAL_LLD_SOURCE_DIR=../lld -DLLVM_EXTERNAL_CLANG_SOURCE_DIR=../clang ../llvm
make -j
sudo make install
```
* Build Rocm device library
```
export PATH=/opt/rocm/llvm/bin:$PATH
git clone -b amd-stg-open https://github.com/RadeonOpenCompute/ROCm-Device-Libs.git
cd ROCm-Device-Libs
mkdir -p build && cd build
CC=clang CXX=clang++ cmake -DLLVM_DIR=/opt/rocm/llvm -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_WERROR=1 -DLLVM_ENABLE_ASSERTIONS=1 ..
make -j
sudo make install
```
* Build HIP
```
git clone -b master https://github.com/ROCm-Developer-Tools/HIP.git
cd HIP
mkdir -p build && cd build
cmake -DCMAKE_INSTALL_PREFIX=/opt/rocm/hip -DHIP_COMPILER=clang -DCMAKE_BUILD_TYPE=Release ..
make -j
sudo make install
```
* Default paths and environment variables:
* By default HIP looks for HSA in /opt/rocm/hsa (can be overridden by setting HSA_PATH environment variable)
* By default HIP is installed into /opt/rocm/hip (can be overridden by setting HIP_PATH environment variable).
* By default HIP looks for clang in /opt/rocm/llvm/bin (can be overridden by setting HIP_CLANG_PATH environment variable)
* By default HIP looks for device library in /opt/rocm/lib (can be overriden by setting DEVICE_LIB_PATH environment variable).
* Optionally, consider adding /opt/rocm/bin to your PATH to make it easier to use the tools.
* Optionally, set HIPCC_VERBOSE=7 to output the command line for compilation to make sure clang is used instead of hcc.
## HIP-nvcc
* Add the ROCm package server to your system as per the OS-specific guide available [here](https://rocm.github.io/ROCmInstall.html#installing-from-amd-rocm-repositories).
* Install the "hip-nvcc" package. This will install CUDA SDK and the HIP porting layer.
```
apt-get install hip-nvcc
```
* Default paths and environment variables:
* By default HIP looks for CUDA SDK in /usr/local/cuda (can be overriden by setting CUDA_PATH env variable)
* By default HIP is installed into /opt/rocm/hip (can be overridden by setting HIP_PATH environment variable).
* Optionally, consider adding /opt/rocm/bin to your path to make it easier to use the tools.
## Verify your installation
Run hipconfig (instructions below assume default installation path) :
```shell
/opt/rocm/bin/hipconfig --full
```
Compile and run the [square sample](https://github.com/ROCm-Developer-Tools/HIP/tree/master/samples/0_Intro/square).
# Building HIP from source
HIP source code is available and the project can be built from source on the HCC platform.
1. Follow the above steps to install and validate the binary packages.
2. Download HIP source code (from the [GitHub repot](https://github.com/ROCm-Developer-Tools/HIP).)
3. Install HIP build-time dependencies using ```sudo apt-get install libelf-dev```.
4. Build and install HIP (This is the simple version assuming default paths ; see below for additional options.)
By default, HIP uses HCC to compile programs. To use HIP-Clang, add -DHIP_COMPILER=clang to cmake command line.
```
cd HIP
mkdir build
cd build
cmake ..
make
make install
```
* Default paths:
* By default cmake looks for hcc in /opt/rocm/hcc (can be overridden by setting ```-DHCC_HOME=/path/to/hcc``` in the cmake step).*
* By default cmake looks for HSA in /opt/rocm/hsa (can be overridden by setting ```-DHSA_PATH=/path/to/hsa``` in the cmake step).*
* By default cmake installs HIP to /opt/rocm/hip (can be overridden by setting ```-DCMAKE_INSTALL_PREFIX=/where/to/install/hip``` in the cmake step).*
Here's a richer command-line that overrides the default paths:
```shell
cd HIP
mkdir build
cd build
cmake -DHSA_PATH=/path/to/hsa -DHCC_HOME=/path/to/hcc -DCMAKE_INSTALL_PREFIX=/where/to/install/hip -DCMAKE_BUILD_TYPE=Release ..
make
make install
```
* After installation, make sure HIP_PATH is pointed to `/where/to/install/hip`.
#!/usr/bin/env groovy
// Generated from snippet generator 'properties; set job properties'
properties([buildDiscarder(logRotator(
artifactDaysToKeepStr: '',
artifactNumToKeepStr: '',
daysToKeepStr: '',
numToKeepStr: '10')),
disableConcurrentBuilds(),
parameters([booleanParam( name: 'push_image_to_docker_hub', defaultValue: false, description: 'Push hip & hcc image to rocm docker-hub' )]),
[$class: 'CopyArtifactPermissionProperty', projectNames: '*']
])
////////////////////////////////////////////////////////////////////////
// -- AUXILLARY HELPER FUNCTIONS
////////////////////////////////////////////////////////////////////////
// Return build number of upstream job
@NonCPS
int get_upstream_build_num( )
{
def upstream_cause = currentBuild.rawBuild.getCause( hudson.model.Cause$UpstreamCause )
if( upstream_cause == null)
return 0
return upstream_cause.getUpstreamBuild()
}
////////////////////////////////////////////////////////////////////////
// Return project name of upstream job
@NonCPS
String get_upstream_build_project( )
{
def upstream_cause = currentBuild.rawBuild.getCause( hudson.model.Cause$UpstreamCause )
if( upstream_cause == null)
return null
return upstream_cause.getUpstreamProject()
}
////////////////////////////////////////////////////////////////////////
// Construct the docker build image name
String docker_build_image_name( )
{
return "build-ubuntu-16.04"
}
////////////////////////////////////////////////////////////////////////
// Construct the relative path of the build directory
String build_directory_rel( String build_config )
{
if( build_config.equalsIgnoreCase( 'release' ) )
{
return "build/release"
}
else
{
return "build/debug"
}
}
////////////////////////////////////////////////////////////////////////
// Lots of images are created above; no apparent way to delete images:tags with docker global variable
def docker_clean_images( String org, String image_name )
{
// Check if any images exist first grepping for image names
int docker_images = sh( script: "docker images | grep \"${org}/${image_name}\"", returnStatus: true )
// The script returns a 0 for success (images were found )
if( docker_images == 0 )
{
// Deleting images can fail, if other projects have built on top of that image and are now dependent on it.
// This should not be treated as a hip build failure. This requires cleanup at a later time, possibly through
// another job
try
{
// Best attempt to run bash script to clean images
// deleting images based on hash seems to be more stable than through name:tag values because of <none> tags
sh "docker images | grep \"${org}/${image_name}\" | awk '{print \$1 \":\" \$2}' | xargs docker rmi"
}
catch( err )
{
println 'Failed to cleanup a few images; probably the images are used as a base for other images'
currentBuild.result = 'SUCCESS'
}
}
}
////////////////////////////////////////////////////////////////////////
// -- BUILD RELATED FUNCTIONS
////////////////////////////////////////////////////////////////////////
// Checkout source code, source dependencies and update version number numbers
// Returns a relative path to the directory where the source exists in the workspace
String checkout_and_version( String platform )
{
String source_dir_rel = "src"
String source_hip_rel = "${source_dir_rel}/hip"
stage("${platform} clone")
{
dir( "${source_hip_rel}" )
{
// checkout hip
checkout([
$class: 'GitSCM',
branches: scm.branches,
doGenerateSubmoduleConfigurations: scm.doGenerateSubmoduleConfigurations,
extensions: scm.extensions + [[$class: 'CleanCheckout']],
userRemoteConfigs: scm.userRemoteConfigs
])
}
}
return source_hip_rel
}
////////////////////////////////////////////////////////////////////////
// This creates the docker image that we use to build the project in
// The docker images contains all dependencies, including OS platform, to build
def docker_build_image( String platform, String org, String optional_build_parm, String source_hip_rel, String from_image )
{
String build_image_name = docker_build_image_name( )
String dockerfile_name = "dockerfile-build-ubuntu-16.04"
def build_image = null
stage("${platform} build image")
{
dir("${source_hip_rel}")
{
def user_uid = sh( script: 'id -u', returnStdout: true ).trim()
// Docker 17.05 introduced the ability to use ARG values in FROM statements
// Docker inspect failing on FROM statements with ARG https://issues.jenkins-ci.org/browse/JENKINS-44836
// build_image = docker.build( "${org}/${build_image_name}:latest", "--pull -f docker/${dockerfile_name} --build-arg user_uid=${user_uid} --build-arg base_image=${from_image} ." )
// JENKINS-44836 workaround by using a bash script instead of docker.build()
sh "docker build -t ${org}/${build_image_name}:latest -f docker/${dockerfile_name} ${optional_build_parm} --build-arg user_uid=${user_uid} --build-arg base_image=${from_image} ."
build_image = docker.image( "${org}/${build_image_name}:latest" )
}
}
return build_image
}
////////////////////////////////////////////////////////////////////////
// This encapsulates the cmake configure, build and package commands
// Leverages docker containers to encapsulate the build in a fixed environment
def docker_build_inside_image( def build_image, String inside_args, String platform, String optional_configure, String build_config, String source_hip_rel, String build_dir_rel )
{
String source_hip_abs = pwd() + "/" + source_hip_rel
build_image.inside( inside_args )
{
stage("${platform} make ${build_config}")
{
// The rm command needs to run as sudo because the test steps below create files owned by root
sh """#!/usr/bin/env bash
set -x
rm -rf ${build_dir_rel}
mkdir -p ${build_dir_rel}
cd ${build_dir_rel}
cmake -DCMAKE_BUILD_TYPE=${build_config} -DCMAKE_INSTALL_PREFIX=staging ${optional_configure} ${source_hip_abs}
make -j\$(nproc)
"""
}
// Cap the maximum amount of testing, in case of hangs
// Excluding hipMultiThreadDevice-pyramid & hipMemoryAllocateCoherentDriver tests from automation; due to its flakiness which requires some investigation
timeout(time: 1, unit: 'HOURS')
{
stage("${platform} unit testing")
{
sh """#!/usr/bin/env bash
set -x
cd ${build_dir_rel}
make install -j\$(nproc)
make build_tests -i -j\$(nproc)
ctest --output-on-failure -E "(hipMultiThreadDevice-pyramid|hipMemoryAllocateCoherentDriver)"
"""
// If unit tests output a junit or xunit file in the future, jenkins can parse that file
// to display test results on the dashboard
// junit "${build_dir_rel}/*.xml"
}
}
// Only create packages from hcc based builds
if( platform.toLowerCase( ).startsWith( 'rocm-' ) )
{
stage("${platform} packaging")
{
sh """#!/usr/bin/env bash
set -x
cd ${build_dir_rel}
make package
"""
// No matter the base platform, all packages have the same name
// Only upload 1 set of packages, so we don't have a race condition uploading packages
if( platform.toLowerCase( ).startsWith( 'rocm-head' ) )
{
archiveArtifacts artifacts: "${build_dir_rel}/*.deb", fingerprint: true
archiveArtifacts artifacts: "${build_dir_rel}/*.rpm", fingerprint: true
}
}
}
}
return void
}
////////////////////////////////////////////////////////////////////////
// This builds a fresh docker image FROM a clean base image, with no build dependencies included
// Uploads the new docker image to internal artifactory
String docker_upload_artifactory( String hcc_ver, String artifactory_org, String from_image, String source_hip_rel, String build_dir_rel )
{
def hip_install_image = null
String image_name = "hip-${hcc_ver}-ubuntu-16.04"
stage( 'artifactory' )
{
println "artifactory_org: ${artifactory_org}"
// We copy the docker files into the bin directory where the .deb lives so that it's a clean build everytime
sh "cp -r ${source_hip_rel}/docker/* ${build_dir_rel}"
// Docker 17.05 introduced the ability to use ARG values in FROM statements
// Docker inspect failing on FROM statements with ARG https://issues.jenkins-ci.org/browse/JENKINS-44836
// hip_install_image = docker.build( "${artifactory_org}/${image_name}:${env.BUILD_NUMBER}", "--pull -f ${build_dir_rel}/dockerfile-hip-ubuntu-16.04 --build-arg base_image=${from_image} ${build_dir_rel}" )
// JENKINS-44836 workaround by using a bash script instead of docker.build()
sh "docker build -t ${artifactory_org}/${image_name} --pull -f ${build_dir_rel}/dockerfile-hip-ubuntu-16.04 --build-arg base_image=${from_image} ${build_dir_rel}"
hip_install_image = docker.image( "${artifactory_org}/${image_name}" )
// The connection to artifactory can fail sometimes, but this should not be treated as a build fail
try
{
// Don't push pull requests to artifactory, these tend to accumulate over time
if( env.BRANCH_NAME.toLowerCase( ).startsWith( 'pr-' ) )
{
println 'Pull Request (PR-xxx) detected; NOT pushing to artifactory'
}
else
{
docker.withRegistry('http://compute-artifactory:5001', 'artifactory-cred' )
{
hip_install_image.push( "${env.BUILD_NUMBER}" )
hip_install_image.push( 'latest' )
}
}
}
catch( err )
{
currentBuild.result = 'SUCCESS'
}
}
return image_name
}
////////////////////////////////////////////////////////////////////////
// Uploads the new docker image to the public docker-hub
def docker_upload_dockerhub( String local_org, String image_name, String remote_org )
{
stage( 'docker-hub' )
{
// Do not treat failures to push to docker-hub as a build fail
try
{
sh """#!/usr/bin/env bash
set -x
echo inside sh
docker tag ${local_org}/${image_name} ${remote_org}/${image_name}
"""
docker_hub_image = docker.image( "${remote_org}/${image_name}" )
docker.withRegistry('https://registry.hub.docker.com', 'docker-hub-cred' )
{
docker_hub_image.push( "${env.BUILD_NUMBER}" )
docker_hub_image.push( 'latest' )
}
}
catch( err )
{
currentBuild.result = 'SUCCESS'
}
}
}
////////////////////////////////////////////////////////////////////////
// -- MAIN
// Following this line is the start of MAIN of this Jenkinsfile
String build_config = 'Release'
String job_name = env.JOB_NAME.toLowerCase( )
// The following launches 3 builds in parallel: rocm-head, rocm-3.0.x and cuda-10.x
parallel rocm_3_0:
{
node('hip-rocm')
{
String hcc_ver = 'rocm-3.0.x'
String from_image = 'ci_test_nodes/rocm-3.0.x/ubuntu-16.04:latest'
String inside_args = '--device=/dev/kfd --device=/dev/dri --group-add=video'
// Checkout source code, dependencies and version files