Project Hub

Mesos
mail # user
DC/OS dynamic reservation dosen't work

From: caogaojin 2016-04-30, 17:49

Hi list,

I’m currently using dynamic reservation mechanism with mesosphere/DCOS 1.7, the opensourced version,
and something wired just happened. I tried to setup a new marathon service from the universe repo,
and i did a slight modification, setting mesos_role to `dev`. i also turned off mesos anthentication
mechenism.

My steps are:

First i tried to reserve 2 cpu and 1743 ram to `dev` role using http endpoint, and it works:
# curl -i -d slaveId=e6d590b1-5f2d-4be9-93fd-2fac364584cf-S11 \
-d resources='[
{
"name": "cpus",
"type": "SCALAR",
"scalar": {
"value": 2.0
},
"role": "dev",
"reservation": {
}

},
{
"name": "mem",
"type": "SCALAR",
"scalar": {
"value": 1024.0

},
"role": "dev",
"reservation": {
}
}
]' -X POST http://10.132.46.81:5050/unreserve

Second, the slave shows following infomation:
# curl http://10.132.46.81/mesos/slaves
reserved_resources_full: {
dev: [
{
name: "cpus",
type: "SCALAR",
scalar: {
value: 2
},
role: "dev",
reservation: { }
},
{
name: "mem",
type: "SCALAR",
scalar: {
value: 1743
},
role: "dev",
reservation: { }
}
]
},

Third, i tried to run nginx using the new marathon service, which i also set the task only accept `dev` resources,

ID: /nginx
...
Resource Roles: dev

4. on marathon ui i just see the waiting status, which means there’s no suitable resource offer, marathon reports:

Insufficient resources for [/nginx] (need cpus=1.0, mem=128.0, disk=0.0, ports=(1 dynamic), available in offer: [id { value: "e6d590b1-5f2d-4be9-93fd-2fac364584cf-O7476" } framework_id { value: "e6d590b1-5f2d-4be9-93fd-2fac364584cf-0006" } slave_id { value: "e6d590b1-5f2d-4be9-93fd-2fac364584cf-S11" } hostname: "10.132.46.84" resources { name: "ports" type: RANGES ranges { range { begin: 8182 end: 32000 } } role: "dev" reservation { } } resources { name: "cpus" type: SCALAR scalar { value: 2.0 } role: "dev" reservation { } } resources { name: "mem" type: SCALAR scalar { value: 1743.0 } role: "dev" reservation { } } resources { name: "ports" type: RANGES ranges { range { begin: 1025 end: 2180 } range { begin: 2182 end: 3887 } range { begin: 3889 end: 5049 } range { begin: 5052 end: 6326 } range { begin: 6328 end: 8079 } range { begin: 8082 end: 8180 } } role: "*" } resources { name: "disk" type: SCALAR scalar { value: 41497.0 } role: "*" } resources { name: "cpus" type: SCALAR scalar { value: 1.0 } role: "*" } resources { name: "mem" type: SCALAR scalar { value: 896.0 } role: "*" } url { scheme: "http" address { hostname: "10.132.46.84" ip: "10.132.46.84" port: 5051 } path: "/slave(1)" }] (mesosphere.mesos.TaskBuilder:marathon-akka.actor.default-dispatcher-19)

can anyone help me with this?

thank you.
GaoJin CAO
ChinaMobile(Suzhou) R&D
Phone：18896723732
Mail：caogaojin at cmss.chinamobile.com

From: caogaojin 2016-04-30, 17:59

Anyone has such experiences? I think it may be a dynamic reservation issue.
In my situation, i turn off mesos authentication mechanism, will this be a problem?

GaoJin CAO
ChinaMobile(Suzhou) R&D
Phone：18896723732
Mail：caogaojin at cmss.chinamobile.com <http://cmss.chinamobile.com/>

From: haosdent 2016-04-30, 18:29

Why you send reserve request to http://10.132.46.81:5050/unreserve, is it
should be "http://10.132.46.81:5050/reserve"?

On Sun, May 1, 2016 at 1:59 AM, caogaojin <[EMAIL PROTECTED]>
wrote:

Best Regards,
Haosdent Huang

From: haosdent 2016-04-30, 18:40

And why you add

```
"reservation": {}
```

in the resource define? Which role you used when start Marathon?
On Sun, May 1, 2016 at 2:28 AM, haosdent <[EMAIL PROTECTED]> wrote:
Best Regards,
Haosdent Huang

From: caogaojin 2016-04-30, 23:57

dear Haosdent,
my bad; actually i use http://10.132.46.81:5050/reserve for reservation, and http://10.132.46.81:5050/unreserve
for releasing resources :)

if i do not add this, mesos will report error:

```
HTTP/1.1 400 Bad Request
Date: Sun, 01 May 2016 01:11:50 GMT
Content-Length: 75

Invalid RESERVE operation: Resource cpus(dev):1 is not dynamically reserved
```
i’m using `dev` role in marathon.

Best Regards.

GaoJin CAO
ChinaMobile(Suzhou) R&D
Phone：18896723732
Mail：caogaojin at cmss.chinamobile.com <http://cmss.chinamobile.com/>

From: Guangya Liu 2016-05-01, 00:57

Gaojin,

Some questions and suggestions here:

1) How did you deploy the nginx with new marathon URI? I did not found that
I can configure the new marathon URI with nginx.

2) Can you please send out the curl output for your master node:
a) curl "http://master-ip:5050/master/slaves
<http://9.21.51.123:5050/master/slaves>" 2>/dev/null|python -m json.tool

b) curl "http://master-ip:5050/master/state
<http://9.21.51.123:5050/master/slaves>" 2>/dev/null|python -m json.tool

3) For open source DC/OS issues, it is suggested to post your question to
[EMAIL PROTECTED] where you can always get instant and professional support.

Thanks,

Guangya
On Sun, May 1, 2016 at 1:49 AM, caogaojin <[EMAIL PROTECTED]>
wrote:

From: haosdent 2016-05-01, 04:19

Sorry for make you confusing before. The offer looks match resource
requirement according to your log, let me reproduce it first.

On Sun, May 1, 2016 at 8:57 AM, Guangya Liu <[EMAIL PROTECTED]> wrote:

Best Regards,
Haosdent Huang

From: haosdent 2016-05-01, 06:04

Hi, @caogaoin. I try in with the master branch of Mesos and Marathon. And
it works fine for me.
```
INFO Finished processing dda04940-55b6-4ba7-a7bb-75c2108f3151-O8. Matched 1
ops after 2 passes. cpus(dev) 2.0; mem(dev) 1024.0; cpus(*) 5.0; mem(*)
29808.0; disk(*) 915273.0; ports(*) 31000->31254,31256->32000 left.
(mesosphere.marathon.core.matcher.manager.impl.OfferMatcherManagerActor:marathon-akka.actor.default-dispatcher-2)
```

Let me list my verify steps to help you find your problem when start your
tasks.

1. start Mesos master and agent, and reserve resources for dev.

```
$ ./bin/mesos-master.sh --work_dir=/tmp/mesos --ip=127.0.0.1

$ ./bin/mesos-slave.sh --work_dir=/tmp/mesos --ip=127.0.0.1
$ curl -i -d slaveId=dda04940-55b6-4ba7-a7bb-75c2108f3151-S0 \
-d resources='[
{
"name": "cpus",
"type": "SCALAR",
"scalar": {
"value": 2.0
},
"role": "dev",
"reservation": {
}

},
{
"name": "mem",
"type": "SCALAR",
"scalar": {
"value": 1024.0

},
"role": "dev",
"reservation": {
}
}
]' -X POST http://127.0.0.1:5050/reserve

2. start Marathon with dev role

```
./bin/start --master 127.0.0.1:5050 --zk zk://10.10.120.61:2181/marathon
```

3. start task

```
$ cat task.json
{
"id": "/test-health",
"cmd": "sleep 200",
"cpus": 1.0,
"mem": 128.0
}

$ curl -X POST -H "Content-type: application/json"
http://127.0.0.1:8080/v2/apps -d @task.json
```

On Sun, May 1, 2016 at 12:18 PM, haosdent <[EMAIL PROTECTED]> wrote:
Best Regards,
Haosdent Huang

From: caogaojin 2016-05-01, 07:02

Hi @Haosdent, i have done your steps, yes, it works; But marathon does not cosume `dev` resources,
it uses default role resource instead:

```
# curl 10.132.46.71:5050/slaves | python -m json.tool
"used_resources_full": [
{
"name": "cpus",
"role": "*",
"scalar": {
"value": 1.0
},
"type": "SCALAR"
},
{
"name": "mem",
"role": "*",
"scalar": {
"value": 128.0
},
"type": "SCALAR"
},
{
"name": "ports",
"ranges": {
"range": [
{
"begin": 31967,
"end": 31967
}
]
},
"role": "*",
"type": "RANGES"
}
],
"version": "0.28.1"
```

@Guangya, so it should not be A DC/OS issue, it related to mesos, or maybe marathon.

Best Regards,

GaoJin CAO
ChinaMobile(Suzhou) R&D
Mail：caogaojin at cmss.chinamobile.com <http://cmss.chinamobile.com/>

From: caogaojin 2016-05-01, 07:04

@Haosdent, by the way, i’m using mesos:0.28.1, not the master branch. and marathon version is 1.1.1

Best Regards,

GaoJin CAO
ChinaMobile(Suzhou) R&D
Mail：caogaojin at cmss.chinamobile.com <http://cmss.chinamobile.com/>

From: haosdent 2016-05-01, 07:21

could you show the result of curl 10.132.46.71:5050/
<http://10.132.46.71:5050/slaves>roles It shows consume "dev" in my side.

On Sun, May 1, 2016 at 3:04 PM, caogaojin <[EMAIL PROTECTED]>
wrote:

Best Regards,
Haosdent Huang

From: Guangya Liu 2016-05-01, 07:33

Not sure if you need add a new parameter default_accepted_resource_roles ,
even if you start marathon with role dev, it can still get resources from
both role dev and *.

Please refer to
https://github.com/mesosphere/marathon/blob/master/docs/docs/command-line-flags.md
for marathon configuration.
- v0.9.0 --default_accepted_resource_roles (Optional. Default: all
roles): Default for the "acceptedResourceRoles"attribute as a
comma-separated list of strings. All app definitions which do not specify
this attribute explicitly use this value for launching new tasks. Examples:
*, production,*, production
On Sun, May 1, 2016 at 3:21 PM, haosdent <[EMAIL PROTECTED]> wrote:

From: caogaojin 2016-05-01, 08:03

@Guangya, I tried with default_accepted_resource_roles option set to `dev`; same result, marathon can not start new tasks, waiting for resources.

@Guangya, @Haosdent, yes, on slave side, slave reports the resource consumed comes from `*` role;
```
[root@localhost ~]# curl http://10.132.46.71:5050/slaves | python -m json.tool
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1091 100 1091 0 0 179k 0 --:--:-- --:--:-- --:--:-- 213k
{
"slaves": [
{
"active": true,
"attributes": {},
"hostname": "10.132.46.71",
"id": "439a851d-aa09-47af-bc68-d5588dc9f5da-S0",
"offered_resources": {
"cpus": 0.0,
"disk": 0.0,
"mem": 0.0
},
"offered_resources_full": [],
"pid": "slave(1)@10.132.46.71:5051",
"registered_time": 1462091019.95277,
"reregistered_time": 1462091019.95438,
"reserved_resources": {
"dev": {
"cpus": 2.0,
"disk": 0.0,
"mem": 1024.0
}
},
"reserved_resources_full": {
"dev": [
{
"name": "cpus",
"reservation": {},
"role": "dev",
"scalar": {
"value": 2.0
},
"type": "SCALAR"
},
{
"name": "mem",
"reservation": {},
"role": "dev",
"scalar": {
"value": 1024.0
},
"type": "SCALAR"
}
]
},
"resources": {
"cpus": 4.0,
"disk": 46055.0,
"mem": 6727.0,
"ports": "[31000-32000]"
},
"unreserved_resources": {
"cpus": 2.0,
"disk": 46055.0,
"mem": 5703.0,
"ports": "[31000-32000]"
},
"used_resources": {
"cpus": 1.0,
"disk": 0.0,
"mem": 128.0,
"ports": "[31994-31994]"
},
"used_resources_full": [
{
"name": "cpus",
"role": "*",
"scalar": {
"value": 1.0
},
"type": "SCALAR"
},
{
"name": "mem",
"role": "*",
"scalar": {
"value": 128.0
},
"type": "SCALAR"
},
{
"name": "ports",
"ranges": {
"range": [
{
"begin": 31994,
"end": 31994
}
]
},
"role": "*",
"type": "RANGES"
}
],
"version": "0.28.1"
}
]
}
```
while, on the master side, when curl http://10.132.46.71:5050/roles, it shows the resource comes from
`dev` role;
```
[root@localhost ~]# curl http://10.132.46.71:5050/roles | python -m json.tool
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 250 100 250 0 0 46772 0 --:--:-- --:--:-- --:--:-- 50000
{
"roles": [
{
"frameworks": [],
"name": "*",
"resources": {
"cpus": 0,
"disk": 0,
"mem": 0
},
"weight": 1.0
},
{
"frameworks": [
"439a851d-aa09-47af-bc68-d5588dc9f5da-0002"
],
"name": "dev",
"resources": {
"cpus": 1.0,
"disk": 0,
"mem": 128.0,
"ports": "[31994-31994]"
},
"weight": 1.0
}
]
}
```

moreover, when i reserve all resources(dynamic reservation) from default role to `dev` role, not a single task can start on that slave, either declaring use `dev` resource or `*` resource.
```
"reserved_resources": {
"dev": {
"cpus": 4.0,
"disk": 0.0,
"mem": 6727.0
}
},
"reserved_resources_full": {
"dev": [
{
"name": "mem",
"reservation": {},
"role": "dev",
"scalar": {
"value": 6727.0
},
"type": "SCALAR"
},
{
"name": "cpus",
"reservation": {},
"role": "dev",
"scalar": {
"value": 4.0
},
"type": "SCALAR"
}
]
},
"resources": {
"cpus": 4.0,
"disk": 46055.0,
"mem": 6727.0,
"ports": "[31000-32000]"
},
"unreserved_resources": {
"cpus": 0.0,
"disk": 46055.0,
"mem": 0.0,
"ports": "[31000-

From: caogaojin 2016-05-01, 08:24

@Haosdent, what does http://10.132.46.71:5050/roles mean? shouldn’t that tell us all resources that framework with `dev` role takes, both from `dev` and `*` role? It doesn’t mean resources are consumed from `dev` role, right?
Best Regards,

GaoJin CAO
ChinaMobile(Suzhou) R&D
Mail：caogaojin at cmss.chinamobile.com <http://cmss.chinamobile.com/>

From: haosdent 2016-05-01, 09:07

hmm, I am still digging more information in marathon. It may delay because
I am not familiar with marathon's code and sick today, but I would inform
you once have result. My quick conclusion is this may be the bug of
marathon when match resource requirement in offers from Mesos Master. Still
under finding the root cause of it.

On Sun, May 1, 2016 at 4:24 PM, caogaojin <[EMAIL PROTECTED]>
wrote:

Best Regards,
Haosdent Huang

From: haosdent 2016-05-01, 10:05

This may be a feature of Marathon. I saw it only filter resources from
those unreserved resources. After change

```
diff --git a/src/main/scala/mesosphere/mesos/TaskBuilder.scala
b/src/main/scala/mesosphere/mesos/TaskBuilder.scala
index efa3406..e750881 100644
+++ b/src/main/scala/mesosphere/mesos/TaskBuilder.scala
@@ -80,7 +80,7 @@ class TaskBuilder(app: AppDefinition,

val resourceMatch =
ResourceMatcher.matchResources(
- offer, app, runningTasks, ResourceSelector(acceptedResourceRoles,
reserved = false))
+ offer, app, runningTasks, ResourceSelector(acceptedResourceRoles,
reserved = true))

build(offer, resourceMatch)
}
```

It would use the dynamic resources we reserved before. I find Marathon
document have these words:

```
### Static Reservations

Dynamic reservations can only be created for unreserved resources. If you
specify an agent's resources to be reserved for a role via the Mesos
`--resources` or `--default_role` flag, these resources cannot be used for
dynamic reservations. In addition, if Marathon is started with the
`--default_accepted_resource_roles` flag specifying a value that does not
contain `*`, your application definition should explicitly specify
`"acceptedResourceRoles": ["*"]` in order to allow usage and reservation of
unreserved cluster resources.
```

You could double check this with Marathon mailing list
https://groups.google.com/forum/?hl=en#!forum/marathon-framework . Thanks a
lot!

On Sun, May 1, 2016 at 5:07 PM, haosdent <[EMAIL PROTECTED]> wrote:
Best Regards,
Haosdent Huang

From: caogaojin 2016-05-02, 01:10

@Haosdent, sorry for delay; i’m not familiar with scala, the compilation takes me some time. you are right, after changing the file you told me, it works. But that’s not what i want, it my situation, marathon should takes both reserved resource and unreserved resource; when using official marathon version, marathon will use static reserved resources (mesos agent started with `--default_role dev` flag, and marathon with `--mesos_role dev`), but it seems marathon did not take static reserved resource as *reserved resource*,

```
Considering unreserved resources with roles {dev}. Not all basic resources satisfied: cpus NOT SATISFIED (1.0 > 0.0), mem NOT SATISFIED (128.0 > 0.0) (mesosphere.mesos.ResourceMatcher$:marathon-akka.actor.default-dispatcher-6)
```

the above shows that marathon only considers *unreserved* resources with role `dev`. However IMO, all resources with role `dev` should be reserved resources, right? i’m quite confused, @haosdent, are resources provided by mesos agent that started with `--default_role dev` flag unreseved resources? i will advise mesosphere/marathon community for more help.

Thanks,

Best Regards,

GaoJin CAO
ChinaMobile(Suzhou) R&D
Mail：caogaojin at cmss.chinamobile.com <http://cmss.chinamobile.com/>

From: haosdent 2016-05-02, 04:01

Actually "unreserved" here in Marathon means Resources which don't
contains ReservationInfo. For static reservation, they didn't
contains ReservationInfo, so they are "unreserved". For dynamic
reservation, they have ReservationInfo, so they are "reserved". This is why
your task works in static reservation while could not work in dynamic
reservation.

On Mon, May 2, 2016 at 9:10 AM, caogaojin <[EMAIL PROTECTED]>
wrote:

Best Regards,
Haosdent Huang

From: caogaojin 2016-05-02, 10:53

thanks @haosdent, here is the explanation https://github.com/mesosphere/marathon/issues/3851#issuecomment-216192331 <https://github.com/mesosphere/marathon/issues/3851#issuecomment-216192331>
Best Regards,

GaoJin CAO
ChinaMobile(Suzhou) R&D
Mail：caogaojin at cmss.chinamobile.com <http://cmss.chinamobile.com/>

Main Capabilities

Open Source Agents

Products

Services

About

Contact