[Tutor] airflow dag

[shubham goyal]

Hello,

See this is the code:


from airflow import DAG
from datetime import datetime,timedelta
default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime.now(),
    'email': ['airflow at airflow.com'],
    'email_on_failure': False,
    'email_on_retry': False
}
MAIN_DAG='check_dag'
dag = DAG(dag_id=MAIN_DAG, default_args=default_args,
schedule_interval=None)

with open(file, "r") as f:
    payload = f.read()  # Reading the json data from a file
    SimpleHttpOperator(  # creating cluster using SimpleHttpOperator
        task_id='cluster_create',
        method='POST',
        http_conn_id='qubole_default',
        # for directing to https://qa.qubole.net/api
        endpoint='/v2/clusters?auth_token=%s' % (passwd),
        data=payload,
        headers={"Content-Type": "application/json"},
        params={'auth_token': passwd},
        response_check=lambda response: True if response.status_code == 200
else False,
        dag=dag
    )


So this code use simplehttpopeator(
https://airflow.incubator.apache.org/code.html) which is used to create the
cluster or post the information on this url(qa.qubole.net/api) this is same
as requests.post() in python.
As you can see i am using passwd here
>    endpoint='/v2/clusters?auth_token=%s' % (passwd),

and i am making many dags like that which are connected and they all use
the password(auth_token) to send the data so i want to pass passwd as
command line argument.

Argument parser is not working here
i tried argument parser and also sys.argv but it doesn't work.

from airflow import DAG
from airflow.operators import SimpleHttpOperator
from datetime import datetime,timedelta
import argparse
import os
parser = argparse.ArgumentParser()
parser.add_argument("passwd")
args = parser.parse_args()
default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime.now(),
    'email': ['airflow at airflow.com'],
    'email_on_failure': False,
    'email_on_retry': False
}
print args.passwd
MAIN_DAG='check_dag'
dag = DAG(dag_id=MAIN_DAG, default_args=default_args,
schedule_interval=None)
file=os.path.join(os.path.dirname(os.path.abspath(__file__)),"check.json")
with open(file, "r") as f:
    payload = f.read()  # Reading the json data from a file
    SimpleHttpOperator(  # creating cluster using SimpleHttpOperator
        task_id='cluster_create',
        method='POST',
        http_conn_id='qubole_default',
        # for directing to https://qa.qubole.net/api
        endpoint='/v2/clusters?auth_token=%s' % (args.passwd),
        data=payload,
        headers={"Content-Type": "application/json"},
        params={'auth_token': args.passwd},
        response_check=lambda response: True if response.status_code == 200
else False,
        dag=dag
    )

if i use this its not able to trigger the dag.

for triggering the dag we use
airflow trigger_dag dagid -r runid

argumentparser work in python
python script.py --passwd passwd   #this work

but this is not a python script

airflow trigger_dag dagid -r runid --passwd passwd

sorry for my english.

On Sun, May 28, 2017 at 9:17 PM, shubham goyal <skgoyal721 at gmail.com> wrote:

> Hello all,
>
>
> You are not understanding my point I understand command line argument
> ,sys.argv[],argparser to pass arguments via command line but I want to do
> these things when I trigger the dag
> And how I trigger the dag
> airflow trigger_dag dagname -s runid
>
> This triggering of dag doesn't support to pass command line argument but I
> want to pass authentication token as command line argument when I trigger
> the dag(you can see in airflow docs) so I am asking is there a way to do it.
>
> I tried argument parser but when I use that it's not able to trigger the
> dag.
>
> I want something like this (in cli)
>
> airflow trigger_dag dagname -r runid --passwd auth_token
>
> And want to access this authtoken in Python script.
>
>
> On May 28, 2017 8:18 PM, "Francois Dion" <francois.dion at gmail.com> wrote:
>
>> My mailbox if full of similar stories: companies dumping airflow on their
>> ETL (or similar) group. Those who knew Python succeeded, those who didn't
>> failed, and some even moved to other companies because they couldn't cope
>> with all this complexity dumped on them all at once.
>>
>> Moral of the story, it is best to learn python first, become good at it,
>> then get into specific tools.
>>
>> And, of course, this is not specific to a relatively straightforward
>> module
>> / application like airflow. I've seen the same with scikit-learn. Same
>> also
>> in other languages. And to make matters worse, the curricula (and books)
>> always starts with a crash course in a specific language, then go on and
>> on
>> about the application, perpetuating the myth that learning the programming
>> language well is totally unimportant.
>>
>> Francois
>>
>> On Sun, May 28, 2017 at 9:46 AM, Alan Gauld via Tutor <tutor at python.org>
>> wrote:
>>
>> > On 28/05/17 04:37, shubham goyal wrote:
>> > > Does anybody have answer?
>> >
>> > You received two answers, both of which asked
>> > you to try something and get back to us for more
>> > information. Did you try printing sys.argv?
>> > What was the result?
>> >
>> > And did you try Peter's argparse code?
>> >
>> > You still haven't explained what your dag is?
>> > What library are you using? It may treat args
>> > in a non standard way, but we can't tell if
>> > you don't give us the information we need to
>> > help you.
>> >
>> > --
>> > Alan G
>> > Author of the Learn to Program web site
>> > http://www.alan-g.me.uk/
>> > http://www.amazon.com/author/alan_gauld
>> > Follow my photo-blog on Flickr at:
>> > http://www.flickr.com/photos/alangauldphotos
>> >
>> >
>> > _______________________________________________
>> > Tutor maillist  -  Tutor at python.org
>> > To unsubscribe or change subscription options:
>> > https://mail.python.org/mailman/listinfo/tutor
>> >
>>
>>
>>
>> --
>> raspberry-python.blogspot.com - www.pyptug.org - www.3DFutureTech.info -
>> @f_dion
>> _______________________________________________
>> Tutor maillist  -  Tutor at python.org
>> To unsubscribe or change subscription options:
>> https://mail.python.org/mailman/listinfo/tutor
>>
>