Another edgemicro.sock 'no such file or directory'...

adeger · 02-05-2019 09:43 AM

I'm hoping someone here has seen or can otherwise help me with this problem. When we try to start a version of the microgateway we get no functionality due to the problem "no such file or directory, access '/home/node/edgemicro.sock'" (the stack trace is below).

My understanding is that this directory is really only used in clustered container setups so I'm first wondering if we even need this (now) and if there's a way to not require it.

Note that I don't know if the message about not having access to apigee-access is a root cause, nor the messages about unset environment variables EDGEMICRO_LOCAL and EDGEMICRO_LOCAL_PROXY nor how or where they should be set.

Things I've tried:

1) Reinstallations of the docker image and

2) Restarting Docker (then restarting container)

3) Creating the 'node' user on the host and chown-ing the directory to that user

4) Starting container as root and another user in the docker group

4a) In all cases above the directory edgemicro.sock is not created

5) Creating an external docker volume mapping (outside->inside) /home/user/edgemicro.sock -> /home/node/edgemicro.sock The directory edgemicro.sock is always created but the app in the contaner fails to start and the 'docker logs' command for the container shows messages like:

Edgemicro seems to be already running.

If the server is not running, it might because of incorrect shutdown of the prevous start.

Try removing /home/node/edgemicro.sock and start again

6) Reinstallations of the docker image and container re-running 🙂

7) Finding and grokking as much salient documentation as seems related...and this forum

Here are some environment facts:

Microgateway version: latest

Docker version 1.13.1, build 07f3374/1.13.1

CentOS Linux release 7.5.1804 (Core), selinux running in Permissive mode

Note that I know just enough Docker to be dangerous and am pretty new to the microgateway too so I have no ego on this matter.

======================== Output from 'docker logs <microgateway container name>' command: ===================

Tue, 05 Feb 2019 15:32:34 GMT apigee Operating without access to apigee-access
2019-02-05T15:32:34.406Z start EDGEMICRO_LOCAL: undefined
2019-02-05T15:32:34.407Z start EDGEMICRO_LOCAL_PROXY: undefined
2019-02-05T15:32:34.408Z microgateway Error: ENOENT: no such file or directory, access '/home/node/edgemicro.sock'
    at Object.fs.accessSync (fs.js:301:11)
    at Gateway.start (/home/node/edgemicro-oneapi/node_modules/edgemicro/cli/lib/gateway.js:28:12)
    at Command.commander.command.option.option.option.option.option.option.option.option.option.option.option.option.option.description.action (/home/node/edgemicro-oneapi/node_modules/edgemicro/cli/cmd.js:243:21)
    at Command.listener (/home/node/edgemicro-oneapi/node_modules/commander/index.js:315:8)
    at emitTwo (events.js:126:13)
    at Command.emit (events.js:214:7)
    at Command.parseArgs (/home/node/edgemicro-oneapi/node_modules/commander/index.js:654:12)
    at Command.parse (/home/node/edgemicro-oneapi/node_modules/commander/index.js:474:21)
    at setup (/home/node/edgemicro-oneapi/node_modules/edgemicro/cli/cmd.js:564:15)
    at Object.<anonymous> (/home/node/edgemicro-oneapi/node_modules/edgemicro/cli/edgemicro:49:21)
    at Module._compile (module.js:653:30)
    at Object.Module._extensions..js (module.js:664:10)
    at Module.load (module.js:566:32)
    at tryModuleLoad (module.js:506:12)
    at Function.Module._load (module.js:498:3)
    at Function.Module.runMain (module.js:694:10)

adeger

Bump this because for a bit more attention

Former Community Member

What version of MG/Docker container are you trying with? Are you using the one shipped by Apigee or using your own?

adeger

This is repackaged container from a third party. However, I'm pretty certain that they haven't modified Apigee's implementation to the point where it would affect the edgemicro.sock implementation. Thus I posted the question here (and also soon with the third party) with the thought that the solution might be provided by, and at the same time help, the community.

Former Community Member

Can you please try with the apigee build

docker pull gcr.io/apigee-microgateway/edgemicro:latest

adeger

Thanks for the feedback @srinandans . So, the above error message turned out to be a red herring.

Actually, the mgw was taking a long time to attempt connection and timeout from the Apigee website for bootstrapping, etc. I didn't see this doing a "docker logs" on the mgw because it was taking a while to generate the timeouts. Instead I fell for the misleading error message above which is thrown almost immediately after (re)start.

The timeouts were cured by 1) specifying DNS servers in docker host /etc/docker/daemon.json ( "dns": ["my.dns.ip.1", "my.dns.ip.2", "8.8.8.8" ] ) and rebooting the host because it had messed up the iptables routing...which is a little bit disconcerting.

alancruz

Hi,

I've just tried this one modifying the daemon.json from .docker also the Docker Desktop Network Settings, enabling the fixed DNS but the issue still there while I try to run my docker image with edgemicro start.
Any other clues for this issue ?

ericschult

Make sure that /home/node exists and is writeable by the ID running edgemicro.

edgemicro.sock is one of , if not *the* first file edgemicro needs to write, so a missing write permission would cause your error.

Definitely do NOT mount edgemicro.sock as a directory. Its a special FIFO socket that should remain local to your container.

alancruz

Hi Eric,

Thanks for your prompt reply. I have been checking the container while running and it just stops with a simple edgemicro init. And sadly, yes as you mentioned, its related to permissions:

...
current nodejs version is v8.15.1
current edgemicro version is 2.5.30
  apigee Operating without access to apigee-access +0ms
failed to init configpath file Error: EACCES: permission denied, open '/home/node/.edgemicro/default.yaml'
...

I am trying to run it with Docker Desktop on a Windows Enterprise 10 computer (using Hyper-V). I have been reading and following thousands of post about this but none of them useful.

Can you please aid me and add suggestions or advices (not possible to switch to a Unix operative system) to allow a docker edgemicro image to write on the shared unit drive ?

Any help will be fully appreciated.

PD: I can run a touch file using sh on the container after failing, but when it comes to run an edgemicro init inside it, says the same to me:

failed to init configpath file Error: EACCES: permission denied, open '/home/node/.edgemicro/default.yaml'

etienned

In a Windows Container you should use Windows folder structures, I have built my own container for a Windows Environment(Windows server 2016), it is working, but still not production ready. It uses Windowservercore image, which is a bit big, so still looking at changing to nanoserver.

alancruz

Hi Etienne,

Thanks for the comment and idea, but the situation here is that I am "consuming" this image and the container running as I do not "own" the code I can not modify it. I am still checking for a solution there...

etienned

Hi Alan,

I ran into the same issue, if you are trying to use any of the images available on-line in a Windows Container it will always fail. If you can't use Hyper-V to switch to a Linux container then you will have to rebuild. Mine is working nicely.

alancruz

Hi Etienne,

Thanks for the feedback, actually I am using Hyper-V with Linux Containers on Windows 10 Enterprise. I can pull the image from the registry with its dependencies and so on (node:alpine, etc), but each time I try to run it fails at "edgemicro start OR init" because it can not write inside the container living on the VM, it seems that writing permissions from the shared C drive are missed

etienned

Sorry for not understanding. So if you want to run this container in a system that is not granting you with Super User privileges then it will fail (Like for example: Openshift). If it is Docker running on your own machine and the default registry location is not allowing you to write `then change the MACHINE_STORAGE_PATH of your Docker installation to where you can write, to see your installation directory type

docker info

and look for the Docker Root Dir (This is where all containers will be created)

alancruz

Hi Etienne,

Once again thanks for your assistance I really appreciate it. So I tried with the environment variable (on env.list file), but same issue even shared the host directory on the network and changing its permissions. These are those values, and yes my user is Super User on this Windows machine of my own (I created the test directory without special characters for testing purposes)

PS C:\Users\test> docker info
Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 6
Server Version: 18.09.3
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: e6b3f5632f50dbc4e9cb6288d911bf4f5e95b18e
runc version: 6635b4f0c6af3810594d2770f662f34ddc15b40d
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.14.104-boot2docker
Operating System: Boot2Docker 18.09.3 (TCL 8.2.1)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 913.3MiB
Name: default
ID: 2X22:E6JC:R4PX:XM37:U3R4:R5WQ:DVBB:RYJ6:VDUF:76AX:5HD2:FZCN
Docker Root Dir: /mnt/sda1/var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
 provider=hyperv
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engin

The --env-file is:

MACHINE_STORAGE_PATH=/c/Users/test
COMPOSE_CONVERT_WINDOWS_PATHS=1
DEBUG=*
EDGEMICRO_ORG=internal
EDGEMICRO_ENV=development
EDGEMICRO_KEY=xxx
EDGEMICRO_SECRET=xxx

etienned

So I assume you are running this on a Hyper-V Manager VM or VirtualBox VM. Can you create any container, like when running the following:

docker run hello-world

I know there is an issue with Windows10 and Hyper-V with file-sharing and volume mounting. Is there any reason why you can't install Docker directly on your laptop?

alancruz

Morgen Etienne,

Hope your day started well. Thanks again, so in reply to this one, yes. I run this with Hyper-V using the MobyLinuxVM (Internal: DockerNAT), the Virtual Switch (External: Ethernet card) and a "default" docker machine correctly running (ACTIVE: *) on this Windows 10 lap.

I can run all those samples perfectly, but each time comes to write to my local shared disk (C) it always says permission denied (all /home/node directories are created ok)

c:\test>docker run --privileged --env-file env.list --rm -p 8000:8080 -it -v //c/test/:/home/node/.edgemicro -w /home/node --name mgw-all registry.oneapi-project.net/microgateway/microgateway:latest edgemicro init
current nodejs version is v8.15.1
current edgemicro version is 2.5.30
  apigee Operating without access to apigee-access +0ms
failed to init configpath file Error: EACCES: permission denied, open '/home/node/.edgemicro/default.yaml'
config initialized to /home/node/.edgemicro/default.yaml

i.e. hello-world: OK

c:\test>docker run hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

i.e. docker.sock: OK

c:\test>docker run --rm -v //var/run:/data alpine ls //data
acpid.pid
acpid.socket
crond.pid
docker
docker.pid
docker.sock
haveged.pid
ntpd.pid
sshd.pid
udhcpc.eth0.pid
utmp
vmblock-fuse

I will be more than happy if there is a hint that can pop-out this one, I am still looking for other ways to solve this (I will be double checking again my Hyper-V, File-Sharing and Volume Mounting).

alancruz

Tried with the current registry from ApiGee, same issue:

c:\test>docker run --privileged --env-file env.list --rm -p 8000:8080 -it -v //c/test/:/home/node/.edgemicro -w /home/node --name mgw-all gcr.io/apigee-microgateway/edgemicro:latest edgemicro init

Log Location: [ /opt/apigee/logs/edgemicro.log ]
cd /opt/apigee && edgemicro start -o internal -e development -k xxx -s xxx -r 8000 -d /opt/apigee/plugins &
current nodejs version is v8.12.0
current edgemicro version is 2.5.30
Fri, 08 Mar 2019 11:59:45 GMT apigee Operating without access to apigee-access
2019-03-08T11:59:45.831Z start EDGEMICRO_LOCAL: undefined
2019-03-08T11:59:45.831Z start EDGEMICRO_LOCAL_PROXY: undefined
Fri, 08 Mar 2019 11:59:45 GMT portastic:test Trying to test port 8000
2019-03-08T11:59:45.837Z microgateway Error: ENOENT: no such file or directory, access '/opt/apigee/edgemicro.sock'
    at Object.fs.accessSync (fs.js:301:11)

alancruz

[ For the record ] Went back to the basics and the ApiGee Edge Microgateway ran on Windows 10 + docker + all the other s---tuff. But now another issue, the .yaml file was wrongly generated:

docker run --rm -P -p 8000:8000 -d --name edgemicro -v /var/tmp:/opt/apigee/logs -e EDGEMICRO_PROCESSES=1 -e EDGEMICRO_ORG=int -e EDGEMICRO_ENV=dev -e EDGEMICRO_KEY=xxx -e EDGEMICRO_SECRET=xxx -e EDGEMICRO_CONFIG=$EDGEMICRO_CONFIG -e SERVICE_NAME=edgemicro --security-opt=no-new-privileges --cap-drop=ALL gcr.io/apigee-microgateway/edgemicro:latest

Container running, but after digging into the logs I found the next:

~/logs $ more edgemicro.log
base64: truncated base64 input
cd /opt/apigee && edgemicro start -o int -e dev -k xxx -s xxx -p 1 -d /opt/apigee/plugins &
current nodejs version is v8.12.0
current edgemicro version is 2.5.30
error reading config from /opt/apigee/.edgemicro/int-dev-config.yaml { YAMLException: the stream contains non-printable characters at line 2, column 1:


    ^
    at generateError (/usr/local/lib/node_modules/edgemicro/node_modules/js-yaml/lib/js-yaml/loader.js:165:10)
    at throwError (/usr/local/lib/node_modules/edgemicro/node_modules/js-yaml/lib/js-yaml/loader.js:171:9)
    at captureSegment (/usr/local/lib/node_modules/edgemicro/node_modules/js-yaml/lib/js-yaml/loader.js:259:7)
    at readPlainScalar (/usr/local/lib/node_modules/edgemicro/node_modules/js-yaml/lib/js-yaml/loader.js:506:3)
    at composeNode (/usr/local/lib/node_modules/edgemicro/node_modules/js-yaml/lib/js-yaml/loader.js:1348:20)
    at readBlockMapping (/usr/local/lib/node_modules/edgemicro/node_modules/js-yaml/lib/js-yaml/loader.js:1009:16)
    at composeNode (/usr/local/lib/node_modules/edgemicro/node_modules/js-yaml/lib/js-yaml/loader.js:1332:12)
    at readDocument (/usr/local/lib/node_modules/edgemicro/node_modules/js-yaml/lib/js-yaml/loader.js:1492:3)
    at loadDocuments (/usr/local/lib/node_modules/edgemicro/node_modules/js-yaml/lib/js-yaml/loader.js:1548:5)
    at load (/usr/local/lib/node_modules/edgemicro/node_modules/js-yaml/lib/js-yaml/loader.js:1569:19)
    at Object.safeLoad (/usr/local/lib/node_modules/edgemicro/node_modules/js-yaml/lib/js-yaml/loader.js:1591:10)
    at IO.loadSync (/usr/local/lib/node_modules/edgemicro/node_modules/microgateway-config/lib/io.js:87:20)
    at Loader.get (/usr/local/lib/node_modules/edgemicro/node_modules/microgateway-config/lib/network.js:95:35)
    at Object.get (/usr/local/lib/node_modules/edgemicro/node_modules/microgateway-config/index.js:16:30)
    at Gateway.start (/usr/local/lib/node_modules/edgemicro/cli/lib/gateway.js:60:16)
    at Command.commander.command.option.option.option.option.option.option.option.option.option.option.option.option.option.description.action (/usr/local/lib/node_modules/edgemicro/cli/cmd.js:243:21)
  name: 'YAMLException',
  reason: 'the stream contains non-printable characters',
  mark:
   Mark {
     name: null,
     buffer: '\u00101�0��8#�\n\u0000',
     position: 10,
     line: 1,
     column: 0 },
  message: 'the stream contains non-printable characters at line 2, column 1:\n    \n    ^' }
/usr/local/lib/node_modules/edgemicro/node_modules/microgateway-config/lib/io.js:91
    throw new Error(err);
    ^


Error: YAMLException: the stream contains non-printable characters at line 2, column 1:


    ^
    at IO.loadSync (/usr/local/lib/node_modules/edgemicro/node_modules/microgateway-config/lib/io.js:91:11)
    at Loader.get (/usr/local/lib/node_modules/edgemicro/node_modules/microgateway-config/lib/network.js:95:35)
    at Object.get (/usr/local/lib/node_modules/edgemicro/node_modules/microgateway-config/index.js:16:30)
    at Gateway.start (/usr/local/lib/node_modules/edgemicro/cli/lib/gateway.js:60:16)
    at Command.commander.command.option.option.option.option.option.option.option.option.option.option.option.option.option.description.action (/usr/local/lib/node_modules/edgemicro/cli/cmd.js:243:21)
    at Command.listener (/usr/local/lib/node_modules/edgemicro/node_modules/commander/index.js:315:8)
    at emitTwo (events.js:126:13)
    at Command.emit (events.js:214:7)
    at Command.parseArgs (/usr/local/lib/node_modules/edgemicro/node_modules/commander/index.js:654:12)
    at Command.parse (/usr/local/lib/node_modules/edgemicro/node_modules/commander/index.js:474:21)
    at setup (/usr/local/lib/node_modules/edgemicro/cli/cmd.js:564:15)
    at Object.<anonymous> (/usr/local/lib/node_modules/edgemicro/cli/edgemicro:49:21)
    at Module._compile (module.js:653:30)
    at Object.Module._extensions..js (module.js:664:10)
    at Module.load (module.js:566:32)
    at tryModuleLoad (module.js:506:12)

Any ideas/help ?, thanks again in advance

etienned

Hi Alan,

When you say going back to basics with Docker + Windows10, are you no longer running on Hyper-V Manager.

The best is to start isolating your issues

So just run the basics, start with this, thus you are removing volume mounting and security options. This should work, then you can start adding the additional parameters.

docker run -P -p 8000:8000 -d --name edgemicro -e EDGEMICRO_DOCKER=1 -e EDGEMICRO_ORG=xxx -e EDGEMICRO_ENV=xxx -e EDGEMICRO_KEY=xxx -e EDGEMICRO_SECRET=xxx -e EDGEMICRO_CONFIG=xxx -e SERVICE_NAME=edgemicro gcr.io/apigee-microgateway/edgemicro:2.5.30

How did you generate your config-file: in Windows10 you can use the following

certutil -f -encode xxx-xxx-config.yaml encodeconf.txt

or to test your conf file in Windows10:

echo %EDGEMICRO_CONFIG% >> %crt2.tmp

certutil -decode -f crt2.tmp C:\Users\ContainerAdministrator\.edgemicro\%EDGEMICRO_ORG%-%EDGEMICRO_ENV%-config.yaml

Just have a look that it then with notepad++ to see if it is valid.

alancruz

Hi Etienne,

Hope your week started well and everything is going good over there.

Once again thanks for your support, so I tried your last command and it runs smoothly, so the default.yaml is created with default values, but when I check the content of the xxx-xxx-config.yaml is totally empty. So I think the problem is more related to the local mounting point on the Windows system (my laptop) and the target mounting point (the one on the running container).

I know that is a mess and quite frustrating to run docker on Windows with Hyper-V and make containers run there.

But, do you have any clue on this one ?. Because, I do not find my yaml config because the mounting point is not working for me:

docker run ... -v //c/myWindowsPath:/home/node/.edgemicro ... gcr.io/apigee-microgateway/edgemicro:latest

alancruz

Hi Etienne,

Hope your week is going really great.

Thanks again for your support following up this thread so finally I came with a solution, quite "amiss" since the beginning :c but I am happy it is working now.

[Problem]: Permissions, Accesses, Source Mounting Issues

[Solution]: Overall Windows + Docker Settings. I add a link to that thread here: https://github.com/docker/docker.github.io/issues/2979

Thanks again and wish you all a great week and weekend in advance.

Greets from Stuttgart,
Alan

Another edgemicro.sock 'no such file or directory' microgateway docker question