Announcement

Collapse
No announcement yet.

Smoothing the dependency process: Caffe and others

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Smoothing the dependency process: Caffe and others



    I have been trying to run a variety of tests with PTS on Ubuntu 18.04 (and Linux Mint before that), and I have found that for a large number of them, the dependencies were not automatically downloaded. For most, I've been able to figure out what I needed and install it to get past the error message, but I've hit a wall with trying to build caffe.

    Code:
                The installer exited with a non-zero exit status.
                ERROR: Missing Header File: caffe/proto/caffe.pb.h
                LOG: ~/.phoronix-test-suite/installed-tests/pts/caffe-1.3.3/install-failed.log

    Looking at the log:

    Code:
    make: protoc: Command not found
    CXX src/caffe/layer.cpp
    Makefile:632: recipe for target '.build-cpu_release/src/caffe/proto/caffe.pb.cc' failed
    make: *** [.build-cpu_release/src/caffe/proto/caffe.pb.cc] Error 127
    make: *** Waiting for unfinished jobs....
    CXX src/caffe/data_reader.cpp
    CXX src/caffe/util/signal_handler.cpp
    In file included from src/caffe/util/signal_handler.cpp:7:0:
    ./include/caffe/util/signal_handler.h:4:10: fatal error: caffe/proto/caffe.pb.h: No such file or directory
     #include "caffe/proto/caffe.pb.h"
              ^~~~~~~~~~~~~~~~~~~~~~~~
    compilation terminated.
    Makefile:575: recipe for target '.build-cpu_release/src/caffe/util/signal_handler.o' failed
    make: *** [.build-cpu_release/src/caffe/util/signal_handler.o] Error 1
    In file included from ./include/caffe/layer.hpp:8:0,
                     from src/caffe/layer.cpp:2:
    ./include/caffe/blob.hpp:9:10: fatal error: caffe/proto/caffe.pb.h: No such file or directory
     #include "caffe/proto/caffe.pb.h"
              ^~~~~~~~~~~~~~~~~~~~~~~~
    compilation terminated.
    Makefile:575: recipe for target '.build-cpu_release/src/caffe/layer.o' failed
    make: *** [.build-cpu_release/src/caffe/layer.o] Error 1
    In file included from ./include/caffe/data_reader.hpp:11:0,
                     from src/caffe/data_reader.cpp:7:
    ./include/caffe/util/db.hpp:7:10: fatal error: caffe/proto/caffe.pb.h: No such file or directory
     #include "caffe/proto/caffe.pb.h"
              ^~~~~~~~~~~~~~~~~~~~~~~~
    compilation terminated.
    Makefile:575: recipe for target '.build-cpu_release/src/caffe/data_reader.o' failed
    make: *** [.build-cpu_release/src/caffe/data_reader.o] Error 1
    I have libprotobuf-dev installed, which I undertstand contain the "protoc" command.

    So, if anyone has suggestions for this specifically, they would be welcome. But more generally, is there a master list of dependencies that I can check prior to attempting installation? It is really annoying to have to go through trial and error so much.







  • #2
    First, what version of PTS are you running? If you are running PTS 7.8~8.0, all of the major dependencies should be covered...

    In regards to Caffe, it's sort of a special case. It has a long list of dependencies that are really not used by any other test profiles and such. Due to that long list of dependencies not often used elsewhere and also not on all Linux distros, I don't cover that for Caffe. But looking at the APT lines within http://openbenchmarking.org/innhold/...09cc8a497f5d6c will basically show you the apt-get commands for the dependencies you need for Caffe.
    Michael Larabel
    http://www.michaellarabel.com/

    Comment


    • #3
      Originally posted by Michael View Post
      First, what version of PTS are you running? If you are running PTS 7.8~8.0, all of the major dependencies should be covered...

      In regards to Caffe, it's sort of a special case. It has a long list of dependencies that are really not used by any other test profiles and such. Due to that long list of dependencies not often used elsewhere and also not on all Linux distros, I don't cover that for Caffe. But looking at the APT lines within http://openbenchmarking.org/innhold/...09cc8a497f5d6c will basically show you the apt-get commands for the dependencies you need for Caffe.
      Thanks I will try that, I didn't realize that the install scripts were so conveniently accessible.
      If I have problems with other test installations (or repeated failures of the tests themselves) my I post them here or should I create a new thread for each one?

      Comment


      • #4
        Originally posted by HaltingDeathtrap View Post

        Thanks I will try that, I didn't realize that the install scripts were so conveniently accessible.
        If I have problems with other test installations (or repeated failures of the tests themselves) my I post them here or should I create a new thread for each one?
        You can post them here. Also, make sure you are using PTS 7.8 or ideally the new 8.0 for best experience.
        Michael Larabel
        http://www.michaellarabel.com/

        Comment


        • #5
          Originally posted by Michael View Post

          You can post them here. Also, make sure you are using PTS 7.8 or ideally the new 8.0 for best experience.
          Sorry, I forgot to confirm that I am using 7.8. Adding the dependencies at the top of the link you posted got me past the above error, but now:
          The installer exited with a non-zero exit status.
          ERROR: collect2: error: ld returned 1 exit status
          LOG: ~/.phoronix-test-suite/installed-tests/pts/caffe-1.3.3/install-failed.log

          Comment


          • #6
            So I tried using 8.0 and I still got the same error. Should I take this up at the Caffe GitHub? If I succeed in installing it separately, do I need to do something special to get PTS to see it?

            Comment


            • #7
              A similar issue is discussed here, and leads me to suspect that the problem has something to do with OpenCV versions:
              https://github.com/BVLC/caffe/issues/2348

              And from my error log:

              Code:
              CXX/LD -o .build-cpu_release/examples/siamese/convert_mnist_siamese_data.bin
              .build-cpu_release/lib/libcaffe.so: undefined reference to `cv::imread(cv::String const&, int)'
              .build-cpu_release/lib/libcaffe.so: undefined reference to `cv::imencode(cv::String const&, cv::_InputArray const&, std::vector<unsigned char, std::allocator<unsigned char> >&, std::vector<int, std::allocator<int> > const&)'
              .build-cpu_release/lib/libcaffe.so: undefined reference to `cv::imdecode(cv::_InputArray const&, int)'
              collect2: error: ld returned 1 exit status
              Makefile:619: recipe for target '.build-cpu_release/tools/upgrade_solver_proto_text.bin' failed
              make: *** [.build-cpu_release/tools/upgrade_solver_proto_text.bin] Error 1
              make: *** Waiting for unfinished jobs....
              .build-cpu_release/examples/cpp_classification/classification.o: In function `main':
              classification.cpp:(.text.startup+0x233): undefined reference to `cv::imread(cv::String const&, int)'
              .build-cpu_release/lib/libcaffe.so: undefined reference to `cv::imencode(cv::String const&, cv::_InputArray const&, std::vector<unsigned char, std::allocator<unsigned char> >&, std::vector<int, std::allocator<int> > const&)'
              .build-cpu_release/lib/libcaffe.so: undefined reference to `cv::imdecode(cv::_InputArray const&, int)'
              collect2: error: ld returned 1 exit status
              Makefile:624: recipe for target '.build-cpu_release/examples/cpp_classification/classification.bin' failed
              make: *** [.build-cpu_release/examples/cpp_classification/classification.bin] Error 1
              .build-cpu_release/lib/libcaffe.so: undefined reference to `cv::imread(cv::String const&, int)'
              .build-cpu_release/lib/libcaffe.so: undefined reference to `cv::imencode(cv::String const&, cv::_InputArray const&, std::vector<unsigned char, std::allocator<unsigned char> >&, std::vector<int, std::allocator<int> > const&)'
              .build-cpu_release/lib/libcaffe.so: undefined reference to `cv::imdecode(cv::_InputArray const&, int)'
              collect2: error: ld returned 1 exit status
              Makefile:624: recipe for target '.build-cpu_release/examples/cifar10/convert_cifar_data.bin' failed
              make: *** [.build-cpu_release/examples/cifar10/convert_cifar_data.bin] Error 1
              .build-cpu_release/lib/libcaffe.so: undefined reference to `cv::imread(cv::String const&, int)'
              .build-cpu_release/lib/libcaffe.so: undefined reference to `cv::imencode(cv::String const&, cv::_InputArray const&, std::vector<unsigned char, std::allocator<unsigned char> >&, std::vector<int, std::allocator<int> > const&)'
              .build-cpu_release/lib/libcaffe.so: undefined reference to `cv::imdecode(cv::_InputArray const&, int)'
              collect2: error: ld returned 1 exit status
              .build-cpu_release/lib/libcaffe.so: undefined reference to `cv::imread(cv::String const&, int)'
              .build-cpu_release/lib/libcaffe.so: undefined reference to `cv::imencode(cv::String const&, cv::_InputArray const&, std::vector<unsigned char, std::allocator<unsigned char> >&, std::vector<int, std::allocator<int> > const&)'
              .build-cpu_release/lib/libcaffe.so: undefined reference to `cv::imdecode(cv::_InputArray const&, int)'
              collect2: error: ld returned 1 exit status
              Makefile:619: recipe for target '.build-cpu_release/tools/convert_imageset.bin' failed
              make: *** [.build-cpu_release/tools/convert_imageset.bin] Error 1
              Makefile:619: recipe for target '.build-cpu_release/tools/caffe.bin' failed
              make: *** [.build-cpu_release/tools/caffe.bin] Error 1
              .build-cpu_release/lib/libcaffe.so: undefined reference to `cv::imread(cv::String const&, int)'
              .build-cpu_release/lib/libcaffe.so: undefined reference to `cv::imencode(cv::String const&, cv::_InputArray const&, std::vector<unsigned char, std::allocator<unsigned char> >&, std::vector<int, std::allocator<int> > const&)'
              .build-cpu_release/lib/libcaffe.so: undefined reference to `cv::imdecode(cv::_InputArray const&, int)'
              collect2: error: ld returned 1 exit status
              Makefile:619: recipe for target '.build-cpu_release/tools/extract_features.bin' failed
              make: *** [.build-cpu_release/tools/extract_features.bin] Error 1
              .build-cpu_release/lib/libcaffe.so: undefined reference to `cv::imread(cv::String const&, int)'
              .build-cpu_release/lib/libcaffe.so: undefined reference to `cv::imencode(cv::String const&, cv::_InputArray const&, std::vector<unsigned char, std::allocator<unsigned char> >&, std::vector<int, std::allocator<int> > const&)'
              .build-cpu_release/lib/libcaffe.so: undefined reference to `cv::imdecode(cv::_InputArray const&, int)'
              collect2: error: ld returned 1 exit status
              Makefile:624: recipe for target '.build-cpu_release/examples/siamese/convert_mnist_siamese_data.bin' failed
              make: *** [.build-cpu_release/examples/siamese/convert_mnist_siamese_data.bin] Error 1
              .build-cpu_release/lib/libcaffe.so: undefined reference to `cv::imread(cv::String const&, int)'
              .build-cpu_release/lib/libcaffe.so: undefined reference to `cv::imencode(cv::String const&, cv::_InputArray const&, std::vector<unsigned char, std::allocator<unsigned char> >&, std::vector<int, std::allocator<int> > const&)'
              .build-cpu_release/lib/libcaffe.so: undefined reference to `cv::imdecode(cv::_InputArray const&, int)'
              collect2: error: ld returned 1 exit status
              Makefile:619: recipe for target '.build-cpu_release/tools/upgrade_net_proto_text.bin' failed
              make: *** [.build-cpu_release/tools/upgrade_net_proto_text.bin] Error 1
              .build-cpu_release/lib/libcaffe.so: undefined reference to `cv::imread(cv::String const&, int)'
              .build-cpu_release/lib/libcaffe.so: undefined reference to `cv::imencode(cv::String const&, cv::_InputArray const&, std::vector<unsigned char, std::allocator<unsigned char> >&, std::vector<int, std::allocator<int> > const&)'
              .build-cpu_release/lib/libcaffe.so: undefined reference to `cv::imdecode(cv::_InputArray const&, int)'
              collect2: error: ld returned 1 exit status
              Makefile:619: recipe for target '.build-cpu_release/tools/compute_image_mean.bin' failed
              make: *** [.build-cpu_release/tools/compute_image_mean.bin] Error 1
              .build-cpu_release/lib/libcaffe.so: undefined reference to `cv::imread(cv::String const&, int)'
              .build-cpu_release/lib/libcaffe.so: undefined reference to `cv::imencode(cv::String const&, cv::_InputArray const&, std::vector<unsigned char, std::allocator<unsigned char> >&, std::vector<int, std::allocator<int> > const&)'
              .build-cpu_release/lib/libcaffe.so: undefined reference to `cv::imdecode(cv::_InputArray const&, int)'
              collect2: error: ld returned 1 exit status
              Makefile:624: recipe for target '.build-cpu_release/examples/mnist/convert_mnist_data.bin' failed
              make: *** [.build-cpu_release/examples/mnist/convert_mnist_data.bin] Error 1
              .build-cpu_release/lib/libcaffe.so: undefined reference to `cv::imread(cv::String const&, int)'
              .build-cpu_release/lib/libcaffe.so: undefined reference to `cv::imencode(cv::String const&, cv::_InputArray const&, std::vector<unsigned char, std::allocator<unsigned char> >&, std::vector<int, std::allocator<int> > const&)'
              .build-cpu_release/lib/libcaffe.so: undefined reference to `cv::imdecode(cv::_InputArray const&, int)'
              collect2: error: ld returned 1 exit status
              Makefile:619: recipe for target '.build-cpu_release/tools/upgrade_net_proto_binary.bin' failed
              make: *** [.build-cpu_release/tools/upgrade_net_proto_binary.bin] Error 1

              Comment


              • #8
                OK, so I am having another problem, this time with running the Numpy benchmark.
                Output of phoronix-test-suite debug-benchmark pts/numpy-1.0.5


                Code:
                Phoronix Test Suite v8.0.0
                
                    Installed:     pts/numpy-1.0.5
                
                
                ========================================
                Numpy Benchmark (Run 1 of 1)
                ========================================
                
                
                Test Run Command: cd /home/james/.phoronix-test-suite/installed-tests/pts/numpy-1.0.5/ && ./numpy  2>&1
                
                Test name :   Avg time ( nanoseconds )
                
                Result Key: #_RESULT_#
                
                
                Template Line: Total avg time (nanoseconds): #_RESULT_#
                
                
                Result Parsing Search Key: "Total avg time (nanoseconds):"
                
                
                Result Line: Total avg time (nanoseconds): 0.00
                
                
                Test Result Parser Returning: 0
                
                
                The test run did not produce a result.

                The logfile did not show anything that gave me clues:

                Code:
                lstsqr:
                hyantes:
                repeating:
                julia:
                multiple_sum:
                allpairs_distances_loops:
                growcut:
                periodic_dist:
                conv:
                local_maxima:
                create_grid:
                pairwise:
                fdtd:
                l2norm:
                diffusion:
                slowparts:
                specialconvolve:
                wdist:
                harris:
                reverse_cumsum:
                grayscott:
                smoothing:
                wave:
                grouping:
                hasting:
                fft:
                cronbach:
                vibr_energy:
                rosen:
                mandel:
                evolve:
                allpairs_distances:
                arc_distance:
                
                
                -------------------
                Total avg time (nanoseconds): 0.00
                I also looked at the installation log, and saw nothing to make me thing that something went wrong there:

                Code:
                numpy-benchmarks-master/
                numpy-benchmarks-master/README.rst
                numpy-benchmarks-master/benchit.py
                numpy-benchmarks-master/benchmarks/
                numpy-benchmarks-master/benchmarks/allpairs_distances.py
                numpy-benchmarks-master/benchmarks/allpairs_distances_loops.py
                numpy-benchmarks-master/benchmarks/arc_distance.py
                numpy-benchmarks-master/benchmarks/conv.py
                numpy-benchmarks-master/benchmarks/create_grid.py
                numpy-benchmarks-master/benchmarks/cronbach.py
                numpy-benchmarks-master/benchmarks/diffusion.py
                numpy-benchmarks-master/benchmarks/evolve.py
                numpy-benchmarks-master/benchmarks/fdtd.py
                numpy-benchmarks-master/benchmarks/fft.py
                numpy-benchmarks-master/benchmarks/grayscott.py
                numpy-benchmarks-master/benchmarks/grouping.py
                numpy-benchmarks-master/benchmarks/growcut.py
                numpy-benchmarks-master/benchmarks/harris.py
                numpy-benchmarks-master/benchmarks/hasting.py
                numpy-benchmarks-master/benchmarks/hyantes.py
                numpy-benchmarks-master/benchmarks/julia.py
                numpy-benchmarks-master/benchmarks/l2norm.py
                numpy-benchmarks-master/benchmarks/local_maxima.py
                numpy-benchmarks-master/benchmarks/lstsqr.py
                numpy-benchmarks-master/benchmarks/mandel.py
                numpy-benchmarks-master/benchmarks/multiple_sum.py
                numpy-benchmarks-master/benchmarks/pairwise.py
                numpy-benchmarks-master/benchmarks/periodic_dist.py
                numpy-benchmarks-master/benchmarks/repeating.py
                numpy-benchmarks-master/benchmarks/reverse_cumsum.py
                numpy-benchmarks-master/benchmarks/rosen.py
                numpy-benchmarks-master/benchmarks/slowparts.py
                numpy-benchmarks-master/benchmarks/smoothing.py
                numpy-benchmarks-master/benchmarks/specialconvolve.py
                numpy-benchmarks-master/benchmarks/vibr_energy.py
                numpy-benchmarks-master/benchmarks/wave.py
                numpy-benchmarks-master/benchmarks/wdist.py
                numpy-benchmarks-master/fmt-bench
                numpy-benchmarks-master/run.py
                patching file run.py

                Comment

                Working...
                X